Attracting new users or business as usual? A case study of converting academic subscription-based journals to open access

https://publons.com/publon/10.1162/qss_a_00126

Scientific journals are the core vehicle for the dissemination of scientific knowledge and are central in the movement to open access. The rationale behind open access is more and better utilization of research, both outside and inside academia, and is increasingly a core element in science policy. The engagement for open access has spawned numerous strategies, most of which are connected to choices taken by the author, such as choosing an open access journal or depositing an article in a repository. However, strategies involving journal owners could be more effective. One such strategy is when publishers convert subscription-based journals to open access, which would leave the researchers’ preference of publishing outlet largely unchanged. Converting a journal is usually a decision made by the owners of the journal on a voluntary basis, but the financial incentives are often weak, as academic journals are a lucrative business for publishers and often a necessary source of income for societies. Much of the current debate about open access revolves around the business model of publishers; however, this is a discussion centered on large international publishers and English language journal flagships. There are many important national journals in local languages that are at the other end of the spectrum. These often rely on subsidies and therefore represent deviations from the narrative of international publishers with excessively high revenues (Larivière, Haustein, & Mongeon, 2015). Such journals have been given less attention in the literature, but play a prominent role in local and national research cultures, particularly within the social sciences and humanities (SSH).

In Norway, governmental mandates and open access policies have provided strong incentives for journals to convert from subscription to open access. The journals covered in my investigation are all within the SSH and part of a subsidizing program offered by Research Council Norway (RCN). The program was initially launched in 1991 with the rationale of securing a scientific infrastructure for research on Norwegian culture and society and in the Norwegian language. From 2017 and onwards, RCN has demanded conversion to open access to be eligible for support, and in 2018 the Ministry of Research and Education launched a program with the aim of filling the financial gap following the corresponding loss of subscription fees. These events created strong incentives to convert to open access for the journals that were dependent on financial support. The policies involved in this setup are concerned not only with the survival of national journals and scientific infrastructure, but argue for the usefulness of knowledge made available to the public, particularly research within SSH. My research aim is therefore to identify some of the effects of converting a journal to open access under these circumstances: Does converting a journal to open access account for increased readership and is it plausible that new audiences are reached?

The objective of the study is not only to identify a general increase in visits in the journals, but to investigate to what extent an increase in readership can be credited to the actual conversion from subscription to open access, and whether opening up the journal has a spillover effect on previously published articles still behind the paywall. To accommodate this, I have chosen a core group of 11 journals which all converted in 2017 and kept their back list closed. A second objective is to investigate whether an increase after a conversion happens at previously subscribing institutions, and thus be able to argue for whether new audiences are reached. The research question for this study is What are the effects of converting subscription-based journals to open access in terms of visits?

The novelty lies in the opportunity to study one of the largest noncommercial systemic conversions of journals to open access, within a short identical time frame, with the same basic funding mechanism and policies and also belonging to the same publisher and thus the same technical infrastructure. The decision to keep the back list closed after conversion further contributes to isolating the effects of converting journals.

The structure of the paper is as follows: First, I review the literature on open access and in particular on journal conversion. This includes how research within SSH fields is connected to the concept of public value. Second, I outline the background for the support of Norwegian SSH journals and how this was set in motion due to more attention to open access. Finally, I present the methodology and empirical results before a concluding discussion.

Open access is connected to the usefulness of research in a very direct way by removing barriers to utilizing the scientific literature. However, this is not only for the benefit of the academic community, but also for readers in the wider society. Because the prospects of value for other sectors is a part of the argument for open access, and specifically a part of the policies involving the journals of this study, it is relevant to discuss briefly the usefulness of the SSH and why and how the policies in question connect these disciplines to open access.

In general, funding of public research has always been justified by the prospects of a positive societal outcome and some kind of return on investment (Sarewitz & Pielke, 2007). Most often “return on investment” is associated with impacts in terms of innovation, products, and markets. Even though research evaluations have concentrated on scientific and economic impacts, the nonscientific and noneconomic impacts are equally important, often referred to as public value (Bozeman & Sarewitz, 2011).

Public value is a softer term and something that escapes the metrics unless covered by economic or other proxies in evaluation programs. In a defense of the value of the social sciences, Brewer issues a warning against the idea of measuring impact by proxies in general; it constitutes a threat to the social sciences because “counting the countable, because the countable can be easily counted, renders the impact illegitimate” (Brewer, 2013, pp. 90–91). Brewer further claims that the public value of the social sciences lies in their ability to shape society through their investigation of it and use of that evidence as premises for the challenges and complex problems of the 21st century. The social sciences should be seen as essential tools for the further development of society, and Brewer proposes a reframing of the value concept into a “new public value of social sciences” (Brewer, 2013, pp. 168–169).

This would require researchers to be more problem-oriented rather than discipline-oriented and to engage in the “wicked problems” of society, for example by engaging in normative issues such as fighting political extremism, lack of trust in authorities, or the undemocratic intrusion in social media networks. The arts and humanities are also under the pressure of economic indicators and metrics in science evaluation, and play a similar role in society as the social sciences. Their value is not reflected in simplistic claims about intrinsic values where arts and humanities research is good for its own sake, nor that the arts and humanities are worth billions of euros in the cultural industry. Their value lies in their influence on a society’s capacity for change and transformation (Benneworth, Gulbrandsen, & Hazelkorn, 2016).

The SSH often have a regional component. A recent study covering seven European countries argues that multilingual publishing in national SSH journals keeps locally relevant research alive with an increased potential for creating impact (Kulczycki, Guns et al., 2020). This aligns with the Helsinki Initiative on Multilingualism in Scholarly Communication (Federation of Finnish Learned Societies et al., 2019) which highlights the importance of language diversity in publishing and dissemination.

The public value impact of the SSH is a useful concept for describing the rationale behind the arrangement of supporting SSH journals and eventually why the arrangement in 2017 made open access mandatory. Open access supports the idea of public value, and functions as an “enhancer” of dissemination and means of obtaining knowledge, not only to the academic community, but to user groups outside academia. The Ministry of Research and Education especially points to user groups such as schools, archives, libraries, museums, and employees within health care, municipalities, public offices, and the private sector as beneficiaries (Ministry of Education, 2017, p. 53).

Open access comes with several interpretations, but it can be achieved in three main ways in accordance with the Budapest Open Access Initiative (BOAI, 2002), the first and most cited of three authoritative declarations at the beginning of the millennium. Green open access means depositing a prior (usually peer-reviewed) version of a published article in a dedicated repository, with free access after an embargo period, usually in the range of 6–24 months (Björk, Laakso et al., 2014). Gold open access means that all articles (publisher’s PDF/version of record) in a journal is immediately free to access on the internet. Expenses for publishing must then be covered by other means than subscription, for example by an article processing charge (APC). Hybrid open access is associated with making a single article immediately freely available in an otherwise subscription-based journal by paying an APC. APCs are common in gold open access journals. Non-APC financed gold open access journals are often called diamond open access, which generally is classified under gold open access. There are other approaches that make the literature freely available, which have led to concepts such as delayed open access (Laakso & Björk, 2013) and bronze open access (Piwowar, Priem et al., 2018). Delayed and bronze are often used interchangeably, but are nevertheless regarded as problematic because they often lack proper licensing and may be removed from the internet at the discretion of the publisher. Lack of proper licensing also blocks usage in commercial services and text and data mining and makes the legality of such use unclear. Even if the status of these types as open access may be disputed, they provide for a significant proportion of freely available literature, especially for users whose primary objective is to read the literature rather than seek benefits in other more advanced ways.

The benefits of open access are often categorized in accordance with the vocabulary of Bozeman and Sarewitz. For example, in an evidence-based review of the advantages of open access, the scientific impact, economic impact, and public value find their counterparts in academic impact, economic impact and societal impact (Tennant, Waldner et al., 2016).

Studies of the economic impact of open access have emphasized possible cost savings at universities and the role of large international publishers and the high margins in the international publishing business (Larivière et al., 2015). A Max Planck Institute white paper argues that there are more than enough funds worldwide to both convert the entire global production of scientific articles to open access and to save money at the same time, even with high average APCs (Schimmer & Geshunhn, 2015). However, the economic impact of open access in terms of innovation has been studied to a lesser extent, and evidence for economic benefits has been called “patchy and diverse,” although there are indications of such benefits (Fell, 2019).

Open access policies often argue for a positive effect in terms of societal impact, but there are few empirical studies on this topic in the review by Tennant et al. The few existing ones focus on Altmetrics and advantages when considering exposure in social media, blogs, and other general media outlets. Altmetrics can also serve as gauges for usage outside academia (Tennant et al., 2016), but there is little evidence for the direct usefulness of academic literature for wider society. However, the principal idea that scientific knowledge should be under as few restrictions as possible, recommends opening the literature for the general audience. Dalrymple (2003, p. 36) has expressed the view that “scientific knowledge in its pure form is a classic public good” and as such should be openly available. A related approach is the taxpayer argument, pointing out the public’s principal rights to access what they have financed (Suber, 2003). Even if there is a lack of empirical studies of nonacademic usage of the literature, there are many user groups who are expected to benefit from open access literature. Zuccala (2009) argues for potential of the academic literature for the “layperson” which can be released by digital access and an increasing understanding of science. Open access can support lifelong learning, medical patients and supporting networks, health advocates, environmentalist organizations, and NGOs (Tennant et al., 2016).

Studies of the academic benefits of open access have traditionally been important for open access advocates, primarily to convince researchers that publishing in open access journals is equally or more beneficial compared to subscription-based journals when considering academic impact. Studies have largely focused on citations and have led to what has been summarized as the open access citation advantage (OACA). SPARC Europe, a higher education membership organization advocating open access, maintained a list of studies up until 2015, a list now adjourned “since the citation advantage evidence has now become far more common knowledge” (SPARC Europe, n.d.). This statement needs to be nuanced; in a large-scale study on the prevalence of open access, Piwowar et al. (2018) find that gold open access literature performed below average in terms of citations, while green and hybrid accounted for a small, but significant OACA. Apparently, the “clear citation advantage” for open access can be interpreted as a disadvantage for gold open access and remains partially contested.

A different aspect of academic benefits is increased readership, where one can find a download advantage in most studies examining the transition to open access. Björk (2017) holds that “there is no question that open access per se increases downloads” and increased readership is considered an intrinsic benefit of open access (Laakso, Solomon, & Björk, 2016). A randomized control study of 3,245 articles in 36 journals found a clear download advantage with a 115% increase in HTML views and a 62% increase in downloads of PDFs (Davis, 2011). A review by Davis and Walters (2011) also concludes with a clear download advantage, whereas a citation advantage remains at best unclear. In two studies of Oxford University Press open access models, findings about a hybrid journal are particularly interesting, as the journal contains both closed and open articles. The study reported a 40% increase in online usage of articles when comparing open access articles and articles still behind the paywall. Although the visits could contain white noise in the form of search engines, the conclusion was that hybrid or optional OA models appear to be showing a significant increase in usage (Bird, 2008, 2010). A study by Piwowar, Priem, and Orr (2019) predicts that by 2025, 70% of all article views will be of the 44% proportion of the literature which is open access, compared to 52% of article views in 2019 on a 31% proportion of open access articles.

Although there is a large body of literature on open access, less attention has been given to studies of converted or “flipped” journals, and none studying national journals in a similar context to that of Norway as far as I have discovered. Even though estimates hold that almost 2,400 journals had converted up until 2011, research on this phenomenon has been scarce (Laakso et al., 2016). Up until 2016 no systematic review of the literature on journal conversion existed, a gap that led to a project proposal by Peter Suber at the library of Harvard in March 2015. The final report was published in 2016 which included commentaries by a panel of experts (Solomon, Laakso et al., 2016) and later published as a review article by the same authors (Laakso et al., 2016).

Their main finding is that there is no single universal approach to conversion that will fit all journals, but rather a range of considerations that dictates the method for each individual journal. The report lays out 15 different journal flipping scenarios grouped on whether the journal applies APCs or not, with the argument that the business model that a journal would base its existence on is an important aspect when considering whether or not to convert a journal.

The decision to convert is usually made by the publisher or the society in charge of the journal, while reasons for converting range from idealistic views of making scholarship more accessible or (more pragmatically) increasing readership or raising the number of submissions and/or their quality. Positioning the journal and enhancing its reputation can also be a motivation for flipping. This is a strategy that finds some support in the literature; the increase in citation rates in terms of journal impact factor was investigated in a set of 171 flipped journals by analyzing their characteristics using Web of Science. The study found that flipping journals had a positive effect on the journal’s impact factor, but no obvious citation advantage at the article level (Momeni, Fraser et al., 2019). Two reports by a BioMed Central publisher found a notable positive effect on citations and the impact factors for five journals that had moved to BioMed Central. The effect was linked to whether the journals were already well established in the Journals Citation Report, one of the most common sources of impact factor metrics (Busch, 2014a, 2014b).

In some cases, an open access business model is seen as more economically viable, or in the case of society journals, used to gain more independence from the publisher (Solomon et al., 2016). Converting a journal is not necessarily a peaceful process; there are cases where the editorial board have abandoned their publisher for not taking steps towards open access. This was the case with Glossa, a journal in linguistics, where the editorial board had come from Lingua (Rooryck, 2017) and Quantitative Science studies (QSS), a journal in the field of scientometrics, where the editorial board came from Journal of Informetrics (Waltman, Larivière et al., 2020). The editorial boards of both journals stepped down when negotiations on open access with their publisher Elsevier failed. Technically, these cases could be classified as something different than journal conversion because the editorial board started new journals and the old journals still exist, but they serve as interesting cases on the power structures in the context of converting a journal. Of the conversion scenarios elaborated in the report by Solomon et al. (2016), the Norwegian initiative is listed as an example under the heading of “National Journal Subsidies” (Solomon et al., 2016, p. 64). However, as will be described in the next section, the financial model chosen for the Norwegian initiative equally resembles the “Consortium or Library Partnership Subsidy.” An example in this category is the Open Library of Humanities (OLH), a university library membership consortium funding OLH’s 27 gold open access journals. There are relevant differences between the Norwegian initiative and the consortium model approach, which lie partly at the policy level; the Norwegian initiative is based on a governmental intervention approach and there is a strong degree of pressure involved that is not present in OLH. In addition, the initiative also upholds subsidies. Stakeholder pressure and monetary subsidies are strong conversion facilitators, but although achieving financial security is of high importance, securing subsidies is, in most cases, not a primary goal of a conversion. The Norwegian initiative is mentioned with interest in the report:

Interestingly, a study found that flipping a journal to open access is not an irreversible process. Matthias, Jahn, and Laakso (2019) found 152 cases of reverse flips from open access to subscription in the last 13 years, where 62% of journals across a range of disciplines at one point converted from subscription to open access and back to subscription. This exploratory study suggests that there are tensions within the scholarly publishing system that need to be resolved for a full transition to open access to happen.

Open access has acquired a strong political position in Norway. The government has emphasized open access in several white papers (Ministry of Education, 2009, 2013) and developed national guidelines for open access (Ministry of Education and Research, 2017). The Research Council of Norway (RCN) acquired an open access policy in 2009 and is also one of the original signatories of Plan S (cOAlition S, 2018). Norway has further focused on consortia negotiations with international publishers to prepare for a transition to open access. These negotiations took place with a considerable political investment by the (then) Norwegian Minister of Research and Education.

An important milestone for this study is a white paper issued in 2017, where the ministry initiated a project with the aim of securing finances for Norwegian journals converting to open access (Ministry of Education, 2017). The project was at that point the latest in a series of initiatives from the Norwegian government, in alignment with the political ambitions for open access over the preceding 10 years and a direct follow-up to RCN’s policy of funding only open access journals from 2017 onward.

The political commitment to open access around 2010 coincided with a significant trend in falling subscription income for Norwegian academic journals. Even though readership for many of the journals in the SSH fields in terms of downloads was on the rise due to electronic publishing, subscription income was dropping. One example is a journal investigated in this study, Tidsskrift for Samfunnsforskning (Journal of Social Research) where the number of subscribing institutions fell from over 900 to below 300 in the period between 1994 and 2012, while the download rate tripled from approximately 6,000 in 2007 to 18,000 in 2011 (Sivertsen, 2013, p. 26). In addition, there was a drop in private subscriptions.

Many of the journals were already in a vulnerable situation before the drop in subscriptions and were thus eligible for financial support from RCN, a type of grant given as baseline funding to approximately 40 journals. RCN commissioned a report to investigate how the two trends could meet and to explore possibilities for making the journals open access and what route would be suitable (Sivertsen, 2013). The report argued for the great potential of open access, as the journals published articles concerned with Norwegian society and culture, written mostly in Norwegian. Some 47% of Norwegian articles within the humanities and 38% within the social sciences are written in Norwegian (Sivertsen, 2013). This also heightened the potential for use outside academia.

As a result, RCN announced in 2014 that from 2017 onward, only open access journals would be included in the funding arrangement, and at the same time RCN offered one-time grants to support the conversion of journals to open access. This effectively forced many of the journals not already open onto an open access platform. It also created a predictable financial challenge, as the grants from RCN covered only 50% of the journals’ expenditures and the already falling subscription income would disappear entirely when the journals converted.

The aforementioned white paper from 2017 launched a pilot project from 2018 to 2021, with the (translated) name “Norwegian Open Journals in Humanities and Social Science” (NOJ-SSH). This was primarily launched to investigate and test new financing models and to rewire the financial streams supporting the journals (Ministry of Education, 2017). The arguments from the report (Sivertsen, 2013) seem to have found resonance in the setup of the project, where the journals should be gold open access without APCs and part of a national consortium funding scheme. The applicants to the NOJ-SSH were subject to an evaluation by a committee consisting of university rectors and deans and was based on quality alone, essentially allocating funds from the top of the quality list for as long as the budget allowed.

In 2020 there were 281 active Norwegian journals in the authoritative journal list maintained by the Norwegian Centre for Research Data (NSD). These are all eligible for the Result Based Financing Scheme (RBO) in Norway, a prerequisite for applying for NOJ-SSH. A total of 41 journals applied to NOJ-SSH and 25 received grants.

The research question chosen for this study asks about the effects of converting journals in terms of visits. Visits are defined as a page views or PDF downloads of articles in the journal, according to the Counter protocol, a protocol used by academic publishers for reporting consistent use of their electronic resources (Project Counter, n.d.). Based on previous studies, one can expect a positive development in the number of visits, which should come in addition to any existing positive trend. Articles still behind the paywall are expected to behave as regular subscription-based journal articles, and not be affected by a conversion to open access, which happens at a later stage. Because users at previously subscribing higher education institutions already had full access, the expectation is that conversion of journals will have no substantial impact on visiting numbers for this user group.

The study therefore proposes the following hypotheses:

• H1: Converting a journal to open access will cause an (additional) increase in visiting numbers for the journal.

• H2: Converting a journal will not cause an (additional) increase in visiting numbers for previously published closed articles (back list).

• H3: Converting a journal will not lead to a substantial (additional) increase in visiting numbers for previous subscribers in the HE sector, compared to other user groups.

The effects on converted journals investigated in this study are limited to the quantifiable effects expressed in visiting data in the period 2014–2019 for a selection of NOJ-SSH journals belonging to the publisher Universitetsforlaget. Other features, such as citation-based metrics, were difficult to obtain, as national native-speaking journals are rarely indexed in commercial indexes such as Web of Science and Scopus. In addition, the relatively short time span involved would not provide valid measures on journal metrics based on citation counts.

The overall strategy was the following:

1. Identify the journals among the NOJ-SSH applicants that had flipped from subscription to open access in 2017 and at the same time kept articles published before conversion closed.

2. Use the time of conversion as an intervention point and identify changes in visiting numbers in the time series, using a Bayesian approach. A selection of subscription-based journals from the same publisher was used as a control.

3. Investigate the visiting numbers for previously subscribing higher education institutions in Norway and compare these with the overall visiting patterns.

Open access journals are either born open access or converted from subscription. The latter group can be divided into two new groups: those that make available the journal’s entire backlog of articles at the time of flipping and those whose previously published articles still are behind the paywall after the time of conversion (Table 1). Among the 41 applicants, all journals except Historisk Tidsskrift were already open access; they were either born open access or converted to open access as part of the RCN scheme effective from 2017. Historisk Tidsskrift was still subscription based at the time of application, but converted in 2018.

At the time of application, all journals were either registered in the Directory of Open Access Journals (DOAJ), had submitted an application to DOAJ, or were planning to. All journals except one used double blind peer review; one used single blind. All journals used some version of the creative commons license, except Mediehistorisk Tidsskrift (Journal of Media History) which was still undecided.

The selection of journals in this study are the journals that kept their back list closed after conversion, and thus provide two distinct methods of access within the same journal.

In cooperation with the publisher Universitetsforlaget, the study obtained access to the journals’ visiting logs. This is Norway’s largest academic publisher, with 75 journals in its portfolio, and with the highest number of both applicants (19) and recipients (16) of grants in NOJ-SSH. All 13 NOJ-SSH applicants with a closed back list are represented by Universitetsforlaget. It supplied three interlinked data sources; (a) monthly number of total visits per journal; (b) yearly number of visits, separated by year of publication; and (c) yearly total visits per journal for each subscribing customer/institution.

The data sources also provided the opportunity for cross-validation and verification of calculations throughout the study.

In addition to having a closed back list, the selection of journals depended on the coverage of a particular set of criteria, where the first four were also NOJ-SSH requirements. All journals should

• be a part of NSD’s authoritative journal list;

• have nonambiguous ISSN, e-ISSN, and title;

• have articles primarily in Norwegian;

• operate within the fields of SSH, according to NSD’s list of channels;

• have complete data on visiting numbers for 2014–2019. This excludes newly started journals, terminated journals or journals recently moved to or acquired from another publisher; and

• have converted from subscription to open access in 2017.

In addition to the group of converted journals (Table 2), a selection of Universitetsforlaget’s subscription-based journals was chosen as a control group. Besides being subscription based, the profile of the journals should be equal to that of the primary group and adhere to the above-listed criteria, except for the last criterion regarding conversion date. Of Universitetsforlaget’s 75 journals, of which 40 are subscription based, 12 journals met the criteria (Table 3).

The initial inspection of visiting data for the two groups of journals showed an almost equal development in average growth from 2014 to 2016, which is up to the point of conversion. The average increase for the groups was calculated by finding the increase in percentage for each journal (respectively from 2014 to 2016 and 2014 to 2019) and then calculating the average of all journals. Both groups of journals had an average increase of approximately 29% from 2014–2016, while the average increase for the whole period showed a larger increase (304.36%) for the group of converted journals. The group of subscription journals had a smaller average increase throughout the whole period (81.03 %), but still accounted for a general tendency of growth in visiting numbers for the journals (see Table 4).

The approach to provide evidence for any causal effects of conversion on visiting numbers was to design a treatment group (converted journals) and a control group (subscription-based journals) and perform the analysis with a Bayesian structural time series model (BSTS). This approach collectively measured any effect of conversion in the group of converted journals.

A BSTS was chosen as it is a suitable method when a randomized control study design is unavailable and a common strategy applied where a synthetic control is needed for prediction (Brodersen, Gallusser et al., 2015). The method is used within different disciplines, such as economics (Poyser, 2019) and medicine (Kurz, Rehm et al., 2019; Vocht, Tilling et al., 2017), while the authors of the method particularly suggest studies of market interventions (Brodersen et al., 2015), which shares important similarities with this study. Examples given are advertising campaigns with the goal of generating more website visits and consequently more downloads and sales of software. The campaign would then act as an intervention and the control group would be defined as the same product in a different market or a very similar product in the same market. The strategy is to synthesize a counterfactual development in the intervention group (had the intervention not taken place) under the assumption that the development would be equal or very similar to the development in the nonintervention group.

The most critical assumption of the model is that the control group is unaffected by the intervention in the treatment group. For example, if there were links or referrals between converted journals and subscription journals that could positively influence visiting numbers in the control group, this would violate this assumption. No such violations were found. The model also assumes that the relationship between covariates and the treated time series remains stable throughout the postperiod, as established during the preperiod. The study could not identify any such violations. An important factor is that the data adheres to the Counter protocol, which accounts for a reasonable degree of stability by removing spikes and sudden boosts created by aggressive search engines etc.

The treatment group was created by adding the visiting numbers in the group of converted journals with a closed back list together, and likewise, the control group was created by adding the visiting numbers in the group of subscription-based journals. These two aggregations were further refined into four different data sets, which are described in this section.

Aggregating visiting numbers of the two groups of journals was a strategy that was deemed appropriate, as the journals shared important similarities. The journals in the treatment group had all converted with some degree of pressure at approximately the same time and all journals in both groups operated within the same scientific disciplines as defined by NOJ-SSH. They are also all handled by the same publisher and served by the same technical platform, so all visiting numbers were thus counted by the same protocol. Visiting numbers included downloads of PDFs and visits to HTML pages and contain little white noise (search engines, crawlers, etc.), as these were removed according to the Code of Practice issued by Project Counter (Project Counter, n.d.).

The aggregation strategy was also justified as the results conceptually resemble what in open access terminology are denoted as megajournals. These are journals that publish articles from a spectrum of disciplines, and as such do not belong exclusively to a specific scientific field. Examples of megajournals are PLOS ONE and Open Library of Humanities.

There are differences in how popular journals are in terms of visits, with some journals accounting for more visits than others. When summarizing visiting numbers, data may be skewed and effects in larger journals may overshadow the effect in smaller journals and thus invalidate the analysis. To control for this, a second data set aggregating the relative increase in visits in percentage from 2014 onwards was created, with the aim of running the same analysis as with the number of visits, with a relative perspective on the same phenomenon.

A third data set was created as a subset of the first aggregation of visiting numbers, but visiting numbers were limited up to the year of flipping and included only data within the period 2014–2016. The data set was used to validate the statistical method by creating a fictitious intervention point in 2015-01 and used to measure the accuracy in the predictions.

The fourth and last data set, also based on the aggregating principle, was limited to visits to articles published before 2017. The purpose was to analyze the effect on visits to articles published before flipping and thus still behind the paywall. This data set differed from the previous data sets by being designed by yearly summaries, rather than monthly.

To summarize the aggregation procedure, the resulting data sets were the following:

1. Set-1: Aggregated visiting numbers

   Result: a synthetic megajournal of flipped journals and a synthetic megajournal of subscription-based journals created by summarizing monthly visiting numbers in the two groups.

2. Set-2: Aggregated relative increase

   Result: a synthetic megajournal of flipped journals and a synthetic megajournal of subscription-based journals created by summarizing relative monthly increase in percentage in the two groups.

3. Set-3: Aggregated visiting numbers, limited to visiting numbers before 2017.

   Result: a synthetic megajournal of flipped journals and a synthetic megajournal of subscription-based journals created by summarizing monthly visiting numbers before 2017 in the two groups.

4. Set-4: Aggregated visiting numbers, limited to articles published before flipping.

   Result: a synthetic megajournal of flipped journals and a synthetic megajournal of subscription-based journals created by summarizing yearly visiting numbers in the two groups limited to articles produced before 2017.

Four separate models were created for each of the four data sets in the following steps.

1. Identify and estimate the effect of conversion by applying the BSTS method to the synthetic (flipped) megajournal with summarized monthly visiting number.

2. Confirm the effect with the same procedure applied to the synthetic (flipped) megajournal with summarized relative increase.

3. Verify the method by applying the BSTS method on the synthetic (flipped) megajournal with summarized monthly visiting number limited up to 2017, with a fictitious intervention date in the period before flipping.

4. Identify and estimate the effect of flipping on the synthetic (flipped) megajournal limited to yearly visiting numbers of articles produced before flipping.

To address the hypothesis about whether the increase can be accounted for by previous subscribing institutions, a separate set of visiting logs for a selection of Norwegian higher education institutions (HEIs) were used. Universitetsforlaget has a consortium agreement with 38 Norwegian institutional subscribers of which 29 are institutions in the HE sector. Twenty-five HE institutions had complete data covering 2014–2019, after adding data from an additional 15 previous members of the consortium which at a later stage merged with one of the current members. The visiting numbers for past members were used to adjust the counts for present members to give a more realistic picture of visiting patterns, as incorporating new users from a merging institution could boost visiting numbers at the time of the merger. The visiting numbers are delineated by the institutions’ IP range; any computer connected to an institutional network infrastructure is counted as belonging to that institution. This includes freely available computers at the library or other multiuser access points.

The second step was to limit visiting data to the list of converted journals in Table 2, using the same 11 journals as in the previous leg. The visiting numbers from the selection of HEIs was added together, forming an HEI group, while a non-HEI group was calculated by subtracting the visits by the HEI group from total visits, as examined in the first leg. The non-HEI group is thus characterized by not containing consortia HEI-members and includes all other visits, including from other countries.

All analysis was run in R, using the library CausalImpact, also developed by Brodersen et al. (2015). The functions in the library construct the model and visualize the results in the plots below, along with the reported statistics.

The first section of the plot in Figure 1 has a solid line showing the actual visiting numbers of the intervention group and a dotted line with a blue confidence interval, showing the predicted number of visits had the intervention not taken place. The middle section shows the absolute increase in visiting numbers (i.e., the surplus accommodated by the intervention), while the bottom pane accumulates the response variable “visits.”

An estimate of the causal effect that the intervention had on the response variable (number of visits) showed an increase of +64%, with a 95% confidence interval of [+57%, +71%]. This should be interpreted as the increase in total visits for all 11 journals in the sample. The positive effect observed by the intervention is unlikely to have been caused by random fluctuations. The probability of obtaining this effect by chance is considered very small (Bayesian one-sided tail-area probability p < .001), and the causal effect can be considered statistically significant.

To control for any spurious effect by some journals overshadowing the effects in others, the same analysis was run on the response variable “relative increase.” The actual values of the variable bear little meaning: The idea of the exercise is to treat all journals as equal contributors to the collective effect of the increase and remove the possibility of skewed data invalidating the analysis. The analysis primarily provides evidence as to whether the previous analysis can be trusted.

Figure 2 shows a very similar development to that of Figure 1. The positive effect in relative increase observed by the intervention is unlikely to have been caused by random fluctuations. The probability of obtaining this effect by chance is considered small (Bayesian one-sided tail-area probability p < .001) and the causal effect can be considered statistically significant.

To control for the predictive strength of the model, a test was performed with a modified version of the data set used in the first analysis. The data were limited to only the number of visits before the actual intervention and introduced a fictitious intervention date at 2015-01. The model was run with the same parameters as the first analysis, with the expectation of a nonsignificant result and a prediction of visits close to the actual number of visits.

As Figure 3 and the statistics show, with an acceptable degree of accuracy, that the prediction line follows the actual numbers of visits reasonably well throughout the period, missing the target of actual number of total visits by only two visits. The causal effect of the intervention is, as expected, not statistically significant, with a posterior tail-area probability of p = 0.49.

The final BSTS analysis was used to measure whether conversion of the journal yielded a positive effect in more visits for articles within the same journals that were still behind a paywall (i.e., published before the time of flipping).

As in Figure 1, the first pane of Figure 4 has a solid line showing the actual visiting numbers of the intervention group and a dotted line with a confidence interval showing the counterfactual predicted number of visits. The lines fluctuate less than in the previous results, as the numbers from the data source were arranged as yearly summaries rather than monthly. The middle section shows the absolute increase in visiting numbers (i.e., the surplus accommodated by the intervention), while the bottom pane accumulates the response variable “visits.” It appears that there is a slight increase from 2018 onward with a relative effect of 4.3%; however, this increase is not statistically significant, with a posterior tail-area probability of p = .19 and a posterior probability of a causal effect of 81%.

In the exploratory plot of Figure 5, the two groups are shown for each journal. HEI-visits is shown by the blue line and non-HEIs by the green. Included for reference is the line for total visits (dotted) and the intercept line on the x-axis showing the approximate time of conversion (placed at the beginning of 2017). Note that the y-axis is floating.

Initial inspection of the data shows that all journals largely exhibit the same pattern, with an increase in non-HEI-visits (green line) in the same year the journal converted, whereas the HEI-visits (blue line) appear largely unaffected. The line with HEI-visits appears relatively stable, while non-HEIs have a higher degree of deviance. In particular, the drop for several journals between 2015 and 2016 is unexpected, and remains unexplained after discussions with Universitetsforlaget. A theory could be that the behavior is due to technical issues; institutions may have changed their IP range without it being registered properly in Universitetsforlaget’s systems or search engines may not have been properly excluded, but this remains speculative. We have not been able to explain this behavior, nor confirm that they in fact are deviations.

To provide statistical evidence for the difference in HEI-visits and non-HEI-visits, the mean increase in percentage per year for each journal was calculated. This was done by stepwise calculating the increase in each journal from the previous year in the period 2014–2019 for both groups. This procedure does not take into account the effect of conversion itself, but was deemed sufficient as the aim is to provide evidence for a difference in increase between the two groups. The mean increase for the two groups per journal is presented in Table 5.

Statistics are given by the sign-test, and the analysis was run in R using the rstatix library. The sign test was chosen because the distribution of differences between paired data values were not normally distributed, which is the assumption of the paired t-test. Transformation of data was not sufficient to satisfy this assumption. Further, the data are not symmetrical around the median, which is a requirement for the paired samples Wilcoxon test.

The null hypothesis for the test is that the median of the paired differences equals zero. The median increase in percentage per year for HEI-visits is significantly different from the median increase for non-HEI-visits, p < .001 (Figure 6).

The group of converted journals with a closed back list shows a statistically significant increase in visiting numbers following flipping. Collectively, the increase is estimated to be a 64% increase in the total number of visits. The effect is confirmed when using relative increase as the dependent variable and the method was validated by applying the same method to a limited data set with fictitious intervention data.

The group of converted journals with a closed back list limited to articles published before the time of conversion showed no significant increase in visiting numbers. There was no statistically significant effect of conversion.

The identified general increase in visiting numbers is not accounted for by previously subscribing higher education institutions.

Hypotheses 1, 2, and 3 are therefore confirmed.

In essence, this study provides evidence that the conversion of journals to open access causes an increase in visits for journals, and further that this increase takes place outside previously subscribing HEIs. Although the effect was expected, it is confirmed under conditions where local journals are in need of funds to secure continued operation and where policy mandates exercise heavy pressure. These results correspond with some of the main open access arguments: Open access generates more readership, partly by granting access to users outside academia (Tennant et al., 2016).

The importance of the results lies partly in the strategic choices made to accommodate open access and partly in the consideration of the arguments for the alleged benefits of open access.

First, open access has become an important part of research policies, with many strategies to accommodate access to the literature for the wider research community and for society in general. Strategies range from local initiatives at institutional libraries up to national mandates and international collaboration between research councils. Strategies for changing the profile of the journal, rather than changing the researcher’s behavior, therefore serve as interesting additions to the policymaking toolbox. This is the case with Plan S, which has issued a report on conversion of journals (Wise & Estelle, 2019) or recent initiatives in Finland on sustainable transitions to open access (Ilva, 2018). This study has the ambition of making a contribution to our understanding of this toolbox.

Second, the study has in general been motivated by claims about the economic, societal, and academic benefits of open access and whether these can be justified. Clearly, any economic benefits of open access are less relevant in this context. The journals are depended on baseline funding and thus at the other end of the income scale compared to large international publishers (Larivière et al., 2015). Hence, primarily this study contributes to the discussion about whether it is likely that open access facilitates societal and academic impact by increased readership and new user groups.

The results are consistent with and confirm the general visits and downloads advantages found in previous studies (Davis, 2011; Davis & Walters, 2011; Piwowar et al., 2019) and extends this research with knowledge about where the effect takes place. The results also expand on the flipping scenarios described in the review on journal conversion, particularly the scenarios of “National Journal Subsidies” and “Consortium or Library Partnership Subsidy” (Laakso et al., 2016; Solomon et al., 2016) which has motivated this study. The report by Solomon et al. was not intended as an analysis of the effects of flipping, but rather on how conversion takes place. This study connects the general visits and downloads advantage with the review’s scenarios and methods for journal conversion.

There are several limitations to the study. The unit of data in this study is “visits” but it remains a challenge to understand what “visits” is ultimately a proxy for. What is sought by authors and journals is readership and impact, but there is at best an unclear link between visits and readership, let alone impact. Visits is a volatile concept and may include everything from a 2-second glance at a webpage to downloading a PDF and thorough reading. If converting a journal to open access and a subsequent rise in visits primarily results in the former type of behavior, then the advantages of conversion are clearly diminished. However, one can assume a correlation between the number of visits and actual readership, albeit not a one-to-one relationship.

Further, visits are essentially counts of requests from computers identified by their IP address, so there is need for additional assumptions to conclude on the origin of visits. The Counter protocol is the de facto standard for counting visits in academic journals, but, naturally it does not reveal who sits in front of the computer or how many users the computer serves. Before journal conversion, HEI users had access to the journals exclusively by institutional subscription schemes. After conversion, the users could also access the literature from other computers, including at home. Likewise, nonacademic users had the opportunity to access the journals from campus computers by visiting the library physically and thus be counted as HEI-visits. In addition, academic users in other Scandinavian countries are not accounted for in the non-HEI-visits group; a small part of the increase can most likely be attributed to them, because language similarities make it relatively easy to read Norwegian literature.

These factors are probable sources of error and make it difficult to translate visits directly into readership, to draw conclusions about the effect-size of visits, or to estimate precisely the types of users, their numbers, and the share of visits. However, even if the extent of these factors is unknown, it can be argued that it is highly unlikely that they account for the increase per se: The 64% increase is too large to be explained by these factors alone.

The investigation has neither been able to identify whether as a general rule all open articles in a journal are visited more often, or if open access accommodates a kind of Matthew effect. The increase in visits is measured at the journal level only, but could be skewed at the article level. Articles attracting visits could attract more visits, while articles that do not get attention in the first place remain largely untouched.

A final issue is to what extent the effects found in the study can be extended to other journals. The journals in the study are all local journals in the same disciplinary fields and in the same subsidiary program; as such, they belong to a rather exclusive selection of journals. The strength of the study design is, in my view, that it accounts for all other variables under the assumption that they influence the journals in the control group and treatment group equally. However, this assumption can also be considered a limitation, as the study design does not reveal unknown influencing variables. These variables may be both properties of the journal or external factors, and as such a generalization to journals outside the selection is not automatically appropriate. For example, the potential of increased readership by converting a journal is most likely intimately connected to the topics of the journals. Very specialized journals in a local language will likely have a much smaller effect.

The limitations of the study are also a good starting point for some of the additional work that needs to be done. Later research might want to look more specifically into users in the private and public sector and whether they represent research organizations or other types of users. New user groups are clearly a part of the motivation for the NOJ-SSH arrangement, which was launched with the vision that a “significant part of research in the Norwegian language would be immediately available for relevant user groups both within and outside academia” (Ministry of Education, 2017, p. 54). This particular policy rests on the concept of public value. Perhaps the most interesting aspect of open access for later research is the usefulness of the literature for the users. How do new users engage with the literature and for what purpose? Do users actively pursue the scientific literature or do they accidently stumble upon it? Later research could also investigate if the literature plays a more prominent role in the public sphere with open access, in terms of not only increased readership in new user groups but increased public value.

I would like to extend my gratitude to Nils Olav Lahlum and Håkon Pagander at Universitetsforlaget for supplying data and offering their expertise. The study has also benefited from insightful advice and comments from Mikael Laakso and from colleagues at the TIK-center and the Osiris research group, particularly Magnus Gulbrandsen. I also thank Henrik Karlstrøm for valuable input regarding coding and two anonymous reviewers for valuable feedback. Nevertheless, the author remains responsible for any mistakes still present.

The author is a PhD candidate at the TIK-center at the University of Oslo while holding a position at UNIT, a governmental body reporting to the Ministry of Education and Research. The position is a part of the department with responsibility for coordinating open access affairs in Norway, which includes the administration of the NOJ-SSH project.

The PhD project is financed by the Research Council Norway (RCN-project: 272456) and is also a part of the Oslo Institute for Research on the Impact of Science (OSIRIS, RCN-project: 256240/O30).

The data set has been made available by agreement with Universitetsforlaget. Data and reproducible code: Wenaas (2021).