Second-order citations in altmetrics: A case study analyzing the audiences of COVID-19 research in the news and on social media

https://www.webofscience.com/api/gateway/wos/peer-review/10.1162/qss_a_00298

Since their inception, altmetrics, in particular those based on social media (Sugimoto, Work et al., 2017) were introduced with the hope of identifying the societal impact of research (Bornmann, 2016; Costas, de Rijcke, & Marres, 2021; Kassab, Bornmann, & Haunschild, 2020; Robinson-Garcia, van Leeuwen, & Ràfols, 2018). The potential of altmetrics for this purpose lies in how they capture the research being circulated by individuals who might not be writing research papers and who can therefore not be observed through traditional bibliometric analyses (Haustein, Bowman, & Costas, 2016). Researchers have been especially interested in identifying nonacademic audiences who share research on social media platforms that are used by a broad segment of society, such as Twitter (now called “X”) and Facebook (Alperin, Gomez, & Haustein, 2019; Costas, Mongeon et al., 2020; Díaz-Faes, Bowman, & Costas, 2019; Haustein, 2019).

Even if a large number of studies have shown that correlations between social media mentions and citations are low (Costas, Zahedi, & Wouters, 2015; Haustein, Peters et al., 2014; Haustein, Costas, & Larivière, 2015), there is little indication that these differences stem from different user populations. That is, most studies of social media user communities find that the activity around research articles on social media is largely generated by academics, not members of the general public (Alperin et al., 2019; Carlson & Harris, 2020; Toupin & Haustein, 2018; Tsou, Bowman et al., 2015; Vainio & Holmberg, 2017). Ferreira, Mongeon, and Costas (2021) found that researchers tend to tweet about subjects similar to those they publish on, indicating that their Twitter activity is an extension of their scholarly communication. Therefore, despite the hope that social media metrics would be able to capture the circulation of research among the public, they seem to reflect online engagement by academic users.

As such, altmetrics in their current form have not been able to deliver on their promise to capture the broader impact of research in society. This expectation might have always been unrealistic, given the diffuse and nonlinear nature of societal impact, but altmetrics have also not been able to provide evidence of when research circulates in the public sphere, nor among whom. Efforts to define what is meant by societal impact and how to measure it invariably point to the circulation of research beyond academic circles, including in places such as policy documents and the news media (e.g., Penfield, Baker et al., 2014; Ravenscroft, Liakata et al., 2017). These suggestions stem from the apparent low volume of research shared on social media by nonacademic audiences (Carlson & Harris, 2020). That is, regardless of the definition and approach sought for understanding societal impact, it will likely be necessary to look beyond the mentions of research outputs themselves and consider other ways and forms in which research circulates among nonacademic audiences.

The news media is a natural place to seek this understanding given their critical role in shaping public discourse (Gallagher, Doroshenko et al., 2021; McCombs, 2002) and informing the public about science (Covens, Kennedy, & Ryan, 2018; Funk, Gottfried, & Mitchell, 2017). Researchers typically write papers with a peer audience in mind, using language that aligns with disciplinary conventions (Fahnestock, 1986) but may be difficult for nonacademic audiences to understand (Fleerackers & Nguyen, 2024). They also tend to focus on the scientific, rather than social, implications of the work (Elliott, 2022b), which may further discourage public engagement with the findings. In contrast, journalists are well attuned to the needs and interests of the public and work to ensure their news coverage is understandable, engaging, and relevant to nonacademic audiences (Kovach & Rosenstiel, 2021). In this way, journalists act as “brokers” of research knowledge, extending its reach by critiquing, contextualizing, and communicating findings in ways that highlight their wider societal significance (Fleerackers, Chtena et al., 2024; Gesualdo, Weber, & Yanovitzky, 2020; Yanovitzky & Weber, 2019). Social media metrics that capture reader and viewer engagement with news coverage have reshaped journalists’ conceptions of and response to their audiences and their interests (Tandoc & Thomas, 2015). Thus, metrics that capture news coverage of research may also offer insights into its circulation in the public spere that have not been captured through other altmetrics (Casino, 2018; Fleerackers, Nehring et al., 2022; Ortega, 2020b).

Although companies such as Altmetric and PlumX now enable researchers and publishers to calculate and track mentions of research in the news media, only a few studies have sought to use these metrics for understanding the audiences of research (e.g., Maggio, Ratcliff et al., 2019; Matthias, Fleerackers, & Alperin, 2020; Moorhead, Krakow, & Maggio, 2021). These studies have yielded insights into the diverse media outlets that report on research and the journalistic approaches they use to do so. As such, they serve as a good starting point for understanding the role of the media in mobilizing research knowledge to a broad, nonacademic public. However, they only examine journalistic attention to the original journal articles, and not the wider attention given to those news stories within society.

One way to address this shortcoming is by exploring so-called “second-order citations,” or social media posts that mention (i.e., cite) web pages that cite research (e.g., news stories that mention research articles) (Priem & Costello, 2010). Unlike typical altmetrics, which focus on “first-order citations” (i.e., social media posts that link to research directly), second-order citations provide the opportunity to observe a common way for nonacademic users to share research with friends or followers (Lemke, Bräuer, & Peters, 2021). Although the impact of these citations is not yet understood, it is evident that news stories mentioning research have the potential to reach users who would not otherwise engage with research on social media, amplifying academic knowledge to broad audiences (Fleerackers, Riedlinger et al., 2022). We seek to investigate these second-order citations through an analysis of tweets and Facebook posts linking the original research articles as well as news stories about the article. How does social media engagement with news stories about research compare to engagement with the research articles themselves with respect to

• the size of the audiences each reaches;

• the nature of the research that is most widely engaged with; and

• the degree of overlap in the accounts and spaces where first- and second-order citations are made.

In March 2021, we identified all COVID-19-related articles published between January 1, 2020 and December 31, 2020 using a well-established set of search terms from the National Library of Medicine (Chen, Allot, & Lu, 2020). We restricted our search to articles from two preprint servers (bioRxiv and medRxiv), because they were prolific sources of COVID-19-related research during the first year of the pandemic, and two peer-reviewed journals (Journal of Virology and British Medical Journal), because they were among the most active networks publishing COVID-19 research and sped up their peer review process in the beginning of the pandemic (Kousha & Thelwall, 2020; Palayew, Norgaard et al., 2020). We limited our study to these four outlets because our intention was to test the viability of the methodology. In total, we identified 3,934 relevant research articles within these four outlets.

We collected news stories that mentioned any of the research articles in our sample by querying the Altmetric Explorer. Altmetric is a company that collects mentions of scholarly documents in online news and social media by regularly scanning the text of thousands of news stories. It identifies news stories that mention research articles either through links to the publication (i.e., through a URL or a publication identifier, such as a DOI) or mentions of study details such as author names, journal titles, and publication dates (Altmetric, 2023). Although these data are not without issues (Ortega, 2019, 2020a), they can be reasonably accurate when working with a fixed set of news outlets (Fleerackers, Nehring et al., 2022).

With this limitation in mind, we restricted our query to five outlets that circulate widely on Twitter and Facebook: BBC, MSN, The New York Times, The Guardian, and The Washington Post, as confirmed by the number of results found when searching Twitter and Facebook for the URLs of websites of the news outlets tracked by Altmetric. Among this sample of outlets, our Altmetric search revealed that 344 (8.7%) of the research articles in our sample were mentioned 1,406 times across 1,221 unique news stories (a news story can mention several different research outputs; see Table 1). On average, each article was mentioned 4.1 times (SD = 6.5).

Although the social media landscape is constantly changing, at the time of the analysis and writing, Twitter and Facebook were the two major platforms where users shared links to news coverage and research articles. Since 2009, Twitter has been recognized as an information network, rather than a social one (Burgess & Baym, 2022), and in June 2022, 66% of all links shared by prominent Twitter accounts directed users to news outlets and wire services (Widjaya & Smith, 2023). In 2022, Facebook was one of the most popular social media platforms in the world, with 2 billion “active” daily users (Mediaweek, 2023) and one of the most used social media platform for accessing news media content (Newman, Fletcher et al., 2022).

To search for social media posts with links to the research articles (i.e., first-order citations), we identified three possible URL types where each research article might be found. The first two URLs were based on Crossref’s guidelines for creating links from an article’s digital object identifier (DOI), using the patterns http://dx.doi.org/{doi} and https://doi.org/{doi} (Hendricks, 2017). A third URL was identified by resolving the DOI URL using a Python script. Similarly, we resolved the shortened URLs provided by Altmetric to arrive at the URL of each of the news stories. To identify social media posts with news stories that mentioned those articles (i.e., second-order citations), we used the URLs provided by Altmetric.

Between March 9 and April 9, 2021, we used Python scripts, including the Twint library (Poldi & Zacharias, 2021), to collect all tweets posted between January 1, 2020 and January 31, 2021 that contained a link to any of the 344 research articles mentioned in the five news outlets or with a link to any of the 1,221 news stories that mentioned those articles. This search yielded 50,299 tweets linking to 325 (94.5%) of the research articles and 97,235 tweets mentioning 486 (39.8%) of the news stories. Tweets returned by the Twint library include both original tweets and replies but exclude both retweets and quote tweets. During the same time period, we used the social media analytics tool Crowdtangle to extract publicly accessible Facebook posts from profiles, groups, and pages (i.e., public Facebook “spaces”; Bruns, Harrington, & Hurcombe, 2020) that contained links to the research articles or news stories. Although public spaces represent only a small proportion of Facebook attention to research (Enkhbayar, Haustein et al., 2019), for ethical reasons, Crowdtangle only provides data related to public community activity. This search yielded 6,420 Facebook posts linking to 246 (71.5%) of the research articles and 14,081 posts linking to 516 (42.3%) of the news stories.

The final data set comprised four elements: 344 research articles; 1,221 news stories that mentioned those research articles; 50,299 tweets and 6,420 Facebook posts that linked to the research articles (i.e., first-order citations); and 97,235 tweets and 14,081 Facebook posts that linked to the news stories (i.e., second-order citations) (Figure 1). Some news stories and social media posts cited more than one research article in our sample, and some social media posts cited more than one news story.

In an effort to estimate whether the share of researchers was higher among Twitter user accounts tweeting research articles in comparison to those users who shared the news stories, we determined the overlap between user accounts in our data set and those identified as researchers in a data set published by Mongeon, Bowman, and Costas (2022, 2023). This data set contains over 400,000 Twitter accounts that are believed to belong to a researcher based on matching names and a record of having linked to one research article with an author sharing the same name. Mongeon et al. (2023)’s approach was limited to identifying accounts that had previously shared their own research, but was demonstrated to have high precision and moderate recall.

All statistics and correlations between first-order citations and second-order citations were calculated using the Python Pandas package (Pandas Development Team, 2023). For each research article, we calculated the sum of Twitter and Facebook posts linking to the article (first-order citations) and compared these totals to the sum of corresponding posts linking to news stories that mentioned that article (second-order citations). For Twitter, we compared the Twitter user IDs of those who shared each research article with those who shared a news story citing that article. Similarly, for Facebook, we compared the account IDs of spaces (i.e., groups, pages, or profiles) where each research article was posted to those where news stories were posted. All Python scripts written to download data, expand URLs, and perform analyses are available online at Alperin (2023b). All data is available at Alperin (2023a).

Our analysis shows that second-order citations (i.e., news stories reporting on the research) are shared more frequently on Twitter and Facebook than first-order citations of the research. Collectively, the news stories written by the five media outlets analyzed were shared approximately twice as often as the research articles and by approximately twice as many unique accounts. The news stories also received approximately twice as much engagement as measured in retweets/shares, likes/reactions, and replies/comments) (Table 2).

High Spearman correlations between Facebook shares and tweets for news (ρ = 0.95) and research articles (ρ = 0.84) show that patterns on both social media platforms were similar (Table 3). However, when comparing first- to second-order citations, the correlations were very low, even when comparing posting about news articles with posting about research articles on the same platform (ρ = 0.13 for tweets and ρ = 0.02 for Facebook posts). Correlations were even lower when calculated for activity across platforms (ρ = −0.01 for first-order Facebook posts and second-order tweets; ρ = 0.00 for first-order tweets and second-order Facebook posts). These relationships are visualized in Figure 2. Each scatterplot represents the relationship between the two variables indicated, with a positive correlation clearly visible for research tweets and Facebook posts (first column, second row) and for news tweets and Facebook posts (bottom row, second column).

Table 4 lists the research articles with the highest number of second-order citations on Twitter and Facebook in comparison to their first-order tweets and Facebook posts. Out of these 13 articles, 11 were preprints when they were mentioned in the news article and only two were already published in journals. However, 10 of the 11 preprints were later published in peer-reviewed journals, including well-established journals such as Science, Nature, Cell, and the New England Journal of Medicine. In line with the correlation patterns described above, only one of the research articles that was highly shared through second-order citations was also highly shared through first-order citations. Articles described findings with obvious practical value, such as assessments of the effectiveness of COVID-19 treatments or prevention measures [1, 2, 9] or situations that may increase one’s risk of contracting the virus [7, 12, 13]. Other highly shared news covered controversial subject matter such as social restrictions and hydroxychloroquine [1, 2]. We further discuss possible factors contributing to the popularity of these articles in Section 4.

As shown in Figure 3, the overlap between the social media accounts that shared first- and second-order citations was very small on both Twitter (14.0% of 27,771 who shared research and 6.4% of 60,296 accounts that shared news stories) and Facebook (22.6% of 3,976 that shared research and 11.0% of 8,191 spaces that shared news stories).

Table 5 lists the top 10 accounts with the most links to research (both first- and second-order citations) together with the top 10 accounts receiving the most engagement (likes and retweets). It includes highly visible scientists (e.g., Eric Topol, Eric Feigl-Ding) and well-known journalists (e.g., Apoorva Mandavilli, George Monbiot), as well as political celebrity figures (i.e., Barack Obama, Hillary Clinton, Joe Biden). The results highlight the drastically different levels of engagement that some social media accounts received. For example, we can see that Barack Obama’s single second-order citation (linking to an article in The New York Times) received 106,785 likes, which is more than all the likes received on tweets from any other account. For comparison, the account of the British Medical Journal (The BMJ) made 379 first-order citations that collectively received 41,068 likes (making it the fifth most liked account).

We then compared the Twitter accounts that had shared research and news with Twitter accounts of known researchers identified in the data set published by Mongeon et al. (2022). We found small overlaps between the 423,920 Twitter accounts identified as belonging to a researcher and the accounts that made first- and second-order citations in our data set, which can be explained by the focus on prioritizing precision over recall of Mongeon et al. (2023)’s approach. However, the share of researchers among accounts with first-order citations (14.0%; n = 3,899) is more than twice as high as that among accounts with second-order citations (6.4%; n = 3,830). Only 718 accounts in the Mongeon et al. (2022) data set were associated with both first- and second-order citations in our data set.

Finally, Table 6 lists the Facebook spaces with the most links to research (both first- and second-order citations) together with the top 10 spaces receiving the most engagement (reactions, shares, and comments). In contrast to the Twitter accounts that shared the most links to research, most of the public Facebook spaces in this list belong to organizations and groups, and not to individuals. These include mainstream media outlets (e.g., The New York Times, The Guardian, The Washington Post), political groups (e.g., Being Liberal, Occupy Democrats), COVID-19 related groups (e.g., Covid-19 / Coronavirus uk, Act Up - Fight Covid-19!), and political celebrity figures (e.g., Bernie Sanders, Hillary Clinton). As with Twitter accounts, there were very different levels of engagement across Facebook spaces. The New York Times, for example, was among those that linked research most frequently and received the most reactions, shares, and comments (similar to the engagement received by posts from The New York Times journalist Apoorva Mandavilli on Twitter). In contrast, Hillary Clinton and Bernie Sanders only linked to research once and twice, respectively, but these posts still put them among the top 11 spaces with the most reactions and comments (similar to and Barack Obama’s posts on Twitter, described above). Other spaces, such as Being Liberal and Occupy Democrats, posted a moderate amount (76th and 447th, respectively) but ranked in the top 10 for all three forms of engagement. For other spaces, such as Act Up - Fight Covid-19! and Coronavirus in Marin, the relationship between posting frequency and engagement was reversed. These spaces posted frequently (7th and 8th) and received moderate engagement (between 120th and 449th).

Numerous studies have found that hyperlinks to research articles shared on social media tend to circulate within closed communities of academic insiders rather than among wider public communities (Alperin et al., 2019; Carlson & Harris, 2020). The results of this study suggest that there may be other overlooked social media audiences who engage with research indirectly by sharing news coverage, or “second-order citations,” of research (Priem & Costello, 2010). Previous studies have suggested that these other audiences may be larger and more representative of society than audiences sharing research directly (Fleerackers, Riedlinger et al., 2022; Lemke et al., 2021). Yet, to our knowledge, this is the first empirical study to document the relative sizes of these two audiences and to explore the degree to which they overlap with one another. The study also introduces a replicable method for identifying and analyzing the reach of second-order citations to research across two social media platforms, paving the way for more altmetrics research to consider indirect engagement with research.

Our findings suggest, albeit from a single case study, that expanding altmetric efforts to consider second-order citations is likely to be a productive avenue for identifying and understanding the circulation of research beyond academic audiences. Our approach complements, and could be powerfully used in conjunction with, previous efforts to expand altmetrics from its origins of simply counting social media engagement with hyperlinks to research articles. In particular, we believe that an examination of second-order citations in conjunction with classification of social media accounts (Costas et al., 2020; Díaz-Faes et al., 2019; Fleerackers, Riedlinger et al., 2022; Ke, Ahn, & Sugimoto, 2017) and examination of social media network characteristics (Alperin et al., 2019; Costas et al., 2021; Robinson-Garcia et al., 2018) has the potential to yield greater insights about the public impact of research than has been possible through the traditional focus on first-order citations.

This promise holds even as we acknowledge that the findings presented here cannot be generalized beyond the specific conditions of this case study of early COVID-19 research communication. Even with this caveat, the differences found between first- and second-order citations—in particular the lower share of researchers among those tweeting news stories rather than original articles and the prevalence of news organizations’ and political celebrities’ posts among those receiving the most engagement—suggest that a broader segment of society may be engaging with research than is commonly captured through first-order citations. In addition, we found that second-order citations also received far more social media engagement. Notably, this reach and engagement advantage was visible even when looking at just a small subset of available news coverage (i.e., stories published by only five media outlets). These findings suggest that online attention to second-order citations would likely be even larger when considering all news stories mentioning research. In other words, our findings indicate that second-order citations can reveal audiences of research that are significantly larger and have limited overlap with audiences identified through first-order citations, which was one of the initial goals of altmetrics (Priem, Taraborelli et al., 2010).

An in-depth discussion of the causes behind the outsized engagement and reach of second-order citations is beyond the scope of this paper. However, potential factors to consider include journalists’ ability to make research knowledge more understandable and relevant to nonacademic audiences (Elliott, 2022a; Gesualdo et al., 2020) and the public’s reliance on news coverage as a key source of science information (Funk et al., 2017)—particularly during the pandemic (Newman, Fletcher et al., 2020). Research into journalistic norms and routines may also help to explain the low correlations and lack of overlap between attention to first- and second-order citations. Specifically, journalistic notions of “relevance” tend to focus on the usefulness or impact of research findings for society, whereas scholars tend to assess relevance based on whether research is novel or significant from an academic perspective (Elliott, 2022a). Relatedly, journalists rely on well-established criteria when deciding what research to cover, selecting findings that have clear importance for their audience; that are surprising, novel, or controversial; or that are likely to spark curiosity and other emotions (Badenschier & Wormer, 2012; Rosen, Guenther, & Froehlich, 2016). In an age of metrics-driven news, journalists also work to maximize the online impact of their work, crafting “catchy” and clickable headlines and monitoring social media trends when selecting and framing stories (Moyo, Mare, & Matsilele, 2019). Collectively, these journalistic practices may facilitate broader engagement with research findings on social media than is typically captured by altmetrics.

The imprint of journalistic selection criteria is visible in the list of highly shared research articles in our study (Table 4). Topics such as treatment or prevention measures or high-risk situations to get infected have obvious practical value, not only aligning with traditional journalistic criteria for identifying “newsworthy” science (Badenschier & Wormer, 2012) but also providing perfect fodder for the “news you can use” style of stories that appeared to be popular early on in the COVID-19 pandemic (Hermida, Varano, & Young, 2022). Other highly shared articles covered controversial topics, which align with journalistic selection criteria. For example, the preprint [1] that reported on the impacts of early nonpharmaceutical interventions on COVID-19 spread received 11,902 second-order citations on Twitter, which might be due to the study’s societal relevance, emotional impact, and politicized topic.

The presence of Barack Obama and other political celebrity figures, well-known journalists, and highly visible scientists in the top rankings (Tables 5 and 6) aligns with previous research that has examined the role of individuals with celebrity status and other elites in the context of diffusing science content on social media (Gallagher et al., 2021; Joubert, Guenther et al., 2023). Given what we know about the important role of influencer accounts in news curation (Bruns, 2018), it is perhaps unsurprising that these accounts generate high engagement from sharing second-order citations, but it is also notable that 11 accounts and six spaces in the top 21 Twitter and Facebook rankings have shared both first- and second-order citations. Although this analysis is not sufficient to make inferences about the role of identity and network position of those involved, it highlights how the identification of second-order citations and their circulation on social media platforms could be used to provide useful qualitative insights into the characteristics and identities of individuals who facilitate social media engagement with research.

Finally, the disconnects between what is popular as a first-order citation and what is popular as a second-order citation (described in Table 3 and shown in Figure 2), and the small overlaps in Twitter accounts and public Facebook spaces that are engaged with both types of citations (described in Figure 3), provide an additional rationale for examining second-order citations more closely. Both of these findings emphasize that a different set of individuals and mechanisms produce second-order citations, at least for the subset of COVID-19 research included in this case study.

Our study design was focused on demonstrating the viability of a methodology for capturing second-order citations and not on offering a comprehensive examination of the phenomenon of second-order citations themselves. As such, the analysis presented has several limitations that prevent us from uncovering the full potential of second-order citations in general, or informing our understanding of societal impact. The first of these is in the choice of COVID-19 as the area of focus and the subsequent choice of both research and news outlets. Each of these choices severely limited the sample size and removed any ability, as acknowledged, to generalize our findings beyond this specific case study. In the case of news outlets, our sample selection was further biased by the U.S. and U.K.-centric choice of outlets. Moreover, our analysis remained limited to high-level aggregate counts of the number of citations received on social media and did not delve into the nature of the engagements, the content of the posts, or the demographic characteristics of those doing the citing. This latter point shows that our approach, as we have presented, shares this limitation with traditional altmetrics. We acknowledge that our method does not inherently address these limitations and would, as with other altmetrics research, need to be expanded further. Finally, recent changes at both Facebook and Twitter have limited the use of their API to researchers and to the public, and have likely jeopardized a direct replication of our method (Lawler, 2022; Weatherbed, 2023). Although the overall approach remains a valid one, any method that relies on commercial platforms is limited by the data made available for analysis.

Although second-order citations were proposed over a decade ago (Priem & Costello, 2010), they remain unexplored by the altmetrics community. The findings presented in this article provide strong evidence for considering these indirect mentions of research as an important measure of the circulation of research beyond the academic community that warrants further exploration. This is timely, given increasing recognition of the societal value of science communication and science journalism (Elliott, 2022a; Gesualdo et al., 2020), not least because of their significant role in mobilizing knowledge during the COVID-19 pandemic (Joubert et al., 2023; Newman et al., 2020).

This paper also makes a significant contribution by pioneering a methodology for compiling and analyzing second-order citation data—something not currently available through existing altmetric data providers. Although further work is required to test, extend, and evolve these methods, the findings described herein give us reason to believe that it will be worthwhile to do so.

The initial idea for this study was jointly conceived by JPA, SH, and Vincent Larivière at the University of Montreal in the summer of 2015, for which we remain grateful 8 years later. We would also like to acknowledge Altmetric.com for access to the news media mentions data. Facebook data was provided by Crowdtangle through the API, with the assistance of Jane Tan at the Digital Media Research Centre, Queensland University of Technology. Finally, we would like to thank the Health Communication Project team at the Scholarly Communication Lab for their input and support in the early stages of the research.

Juan Pablo Alperin: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Software, Supervision, Visualization, Writing—original draft, Writing—review & editing. Alice Fleerackers: Data curation, Formal analysis, Funding acquisition, Methodology, Writing—original draft, Writing—review & editing. Michelle Riedlinger: Data curation, Funding acquisition, Writing—review & editing. Stefanie Haustein: Conceptualization, Funding acquisition, Methodology, Writing—review & editing.

The authors have no competing interests.

This work was supported by the Social Sciences and Humanities Research Council of Canada [435-2020-0401].

The data underlying this study is available at the Harvard Dataverse and cited in the reference list as Alperin (2023a). The software code used to collect and analyze the data is available at Zenodo and cited in the reference list as Alperin (2023b).