The disruption index is biased by citation inflation

https://www.webofscience.com/api/gateway/wos/peer-review/10.1162/qss_a_00333

A measure of disruption was recently developed and applied to empirical citation networks (Funk & Owen-Smith, 2017; Park, Leahey, & Funk, 2023; Wu, Wang, & Evans, 2019). This bibliometric measure, denoted by CD, quantifies the degree to which an intellectual contribution p (e.g., a research publication or invention patent) supersedes the sources cited in its reference list, denoted by the set {r}_p. As defined, CD_p is measured according to the local structure of the subgraph G_p = {r}_p ∪ p ∪ {c}_p comprised of the focal node p, nodes belonging to its reference list {r}_p, and the set of nodes citing either p or any member of {r}_p, denoted by {c}_p. If future intellectual contributions cite p but do not cite members of {r}_p, then it is argued that p plays a disruptive role in the citation network. In what follows we highlight a critical issue underlying the definition of CD which calls into question the utility of this metric and the conclusions that have been drawn following its application to empirical citation networks (Park et al., 2023; Wu et al., 2019). Namely, as the length r_p = ∣{r}_p∣ of the reference list increases, so does the likelihood that one of those papers is highly cited. Consequently, CD_p is a biased measure because reference lists have increased dramatically over time, and so too have the number of citations that highly cited papers accrue (Pan, Petersen et al., 2018)—both phenomena being by-products of citation inflation (Petersen, Pan et al., 2018b), a manifestation of secular growth that naturally extends to patent citation networks (Huang, Chen, & Zhang, 2020; Macher, Rutzer, & Weder, 2024).

Citation inflation (CI) refers to the systematic increase in the number of links introduced to the scientific (or patent) citation network each year. CI is analogous to monetary inflation (Orphanides, 2003; Orphanides & Solow, 1990), whereby as a government prints more money the sticker price of items tends to go up, rendering the impression that the real cost of goods is increasing (to what degree this relationship is valid depends on wage growth and a number of other economic factors). By analogy, it might also be tempting to attribute the increased volume of scientific production to techno-social productivity increases, yet this explanation neglects the persistent growth rate of the inputs (e.g., researchers and research investment) that are fundamental to the downstream production of outputs (e.g., research articles, patents).

Indeed, secular growth underlies various quantities relevant to the study of the scientific endeavor, from national expenditures in R&D to the population size of researchers (Petersen et al., 2018b) and the characteristic number of authors per research publication (Pavlidis, Petersen, & Semendeferi, 2014; Wuchty, Jones, & Uzzi, 2007)—all quantities that have persistently grown over the last century. Nevertheless, the degree to which such growth affects the quantitative evaluation of research outcomes is underappreciated and can manifest in inconsistent measurement designs and metrics. Indeed, the number of citations an article receives is not solely attributable to the novelty or prominence of the research but also depends on the the population size and citing norms of a discipline, and quite fundamentally, the nominal production rate of links in the citation network, among other considerations (Bornmann & Daniel, 2008). Hence, there is real need to distinguish nominal counts versus real values in scientific evaluation, which in the analysis of citation networks requires accounting for when each citation was produced, and in further extensions, how the credit is shared (Huang et al., 2020; Pavlidis et al., 2014; Petersen et al., 2018b; Petersen, Wang, & Stanley, 2010; Shen & Barabàsi, 2014).

So what are the main sources of CI in scientific citation networks and what are the real-world magnitudes of their effects? Figure 1(a) illustrates how CI arises through the combination of longer reference lists, denoted by r(t), compounded by an increasing production volume, denoted by n(t). By way of real-world example, prior calculation of the growth rate of total number of citations produced per year based upon the entire Clarivate Analytics Web of Science citation network estimated that the total volume C(t) ≈ n(t)r(t) of citations generated by the scientific literature grows exponentially, with annual rate g_C = g_n + g_r = 0.033 + 0.018 = 0.051 (Pan et al., 2018). Hence, with the number of links in the citation network growing by roughly 5% annually, the total number of links in the citation network doubles every ln(2) = g_C = 13.6 years!

While the dominant contributor to CI is the growth of n(t) deriving from increased researchers and investment in science coupled with technological advances increasing the rate of manuscript production, the shift away from print towards online-only journals, and the advent of multidisciplinary megajournals (Petersen, 2019), the contribution to CI from growing reference lists alone is nevertheless substantial and varies by discipline (Abt & Garfield, 2002; Dai, Chen et al., 2021; Nicolaisen & Frandsen, 2021; Sánchez-Gil, Gorraiz, & Melero-Fuentes, 2018). By way of example, consider descriptive statistics based upon analysis of millions of research publications comprising the Microsoft Academic Graph (MAG) citation network (Sinha, Shen et al., 2015): in the 1960s, the average (± standard deviation) number of references per articles was $rp¯$ = 9 (±17); by the 2000s, $rp¯$ increased to 23 (±27), a 2.6-fold increase over the 50-year period—see Figure 1(b). This persistent growth in reference list lengths has even occurred in journals with tight space limitations that traditionally publish shorter research articles and letters, such as Nature (Petersen, Arroyave, & Pammolli, 2025).

Meanwhile, as research team sizes—denoted by k_p, and used as a proxy for the production effort associated with a research output—increase in order to address research problems featuring greater topical and methodological breadth, there emerges a nonlinear relationship between $rp¯$ and k_p showing that the modern research article is fundamentally different from those produced even a decade ago—see Figure 1(c). Thus, not only does the nominal value of a citation vary widely by era, but the implications of secular growth on the topology of the citation network and thus citation-based research evaluation are profound (Pan et al., 2018). A standard solution to taming variables that are susceptible to inflation is to use a deflator index, which amounts to normalizing the cross-temporal variation by way of standardized reference point (Petersen et al., 2010, 2018b). A more nefarious problem is the accurate measurement of the quantitative relationship between variables that are independently growing over time, which is susceptible to omitted variable bias if the role of time is neglected in the panel of covariate controls.

In what follows, we demonstrate how CI renders CD unsuitable for cross-temporal analysis using three different approaches. As such, we contribute to a growing number of critiques regarding the validity of CD and the interpretation of trends and correlations (Bentley, Valverde et al., 2023; Bornmann, Devarakonda et al., 2020; Holst, Algaba et al., 2024; Leibel & Bornmann, 2024; Macher et al., 2024; Petersen et al., 2025; Ruan, Lyu et al., 2021; Wu & Wu, 2019), which together call into question the results reported in two Nature cover articles (Park et al., 2023; Wu et al., 2019). To establish how the disruption index suffers from citation inflation and is confounded by shifts in scholarly citation practice, we follow three distinct lines of reasoning: deductive analysis based upon the definition of CD_p, empirical analysis of the Microsoft Academic Graph (MAG) citation network, and computational modeling of synthetic citation networks. In the latter approach, we are able to fully control the sources of the systematic bias underlying CD (namely CI), thereby demonstrating that CD follows a stable frequency distribution in the absence of CI. We conclude with research evaluation and publishing policy implications.

The disruption index is a higher order network metric that incorporates information extending beyond the first-order links connecting to p—those nodes that cite p and are prospective (forward looking or diachronous), and those nodes that are referenced by p, and thus retrospective (backward looking or synchronous) (Glänzel, 2004; Nakamoto, 1988). The original definition of CD was formulated as a conditional sum across the adjacency matrix (Funk & Owen-Smith, 2017), and was subsequently reformulated as a ratio (Wu et al., 2019). According to the latter conceptualization, calculating CD_p involves first identifying three non-overlapping subsets of citing nodes, {c}_p = {c}_i ∪ {c}_j ∪ {c}_k, of sizes N_i, N_j, and N_k, respectively—see Figure 2(a) for a schematic illustration.

To summarize, we argue that a simple deductive explanation trumps the alternative socio-technical explanations offered (Kozlov, 2023; Park et al., 2023) for the decline in CD calculated for publications and patents. In short, we argue that the disruption index CD_p systematically declines for the simple reason that CD features a numerator that is bounded and a denominator that is unbounded. As elaborated in the next section, the term R_k in the denominator is susceptible to CI, and continues to inflate according to two mechanisms: (a) It grows proportional to the reference list length; and (b) it is highly sensitive to highly cited papers, which in the present day can readily achieve hundreds of citations within 5 years, thereby causing R_k to explode and CD to converge to 0 over time.

In this section we show empirically that CD_p declines over time due to the runaway growth of R_k(t) and, implicitly, r(t). While our results are based upon a single representation of the scientific citation network made openly available by the MAG project (Sinha et al., 2015), the implications are generalizable to other citation networks featuring CI, such as patent citation networks (Huang et al., 2020). The citation network we analyzed is formed from the roughly 29.5 × 10⁶ million research articles in the MAG data set that have a digital object identifier (DOI), and were published between 1945 and 2012. To be consistent with Park et al. (2023) and Wu et al. (2019), we calculate CD_p,CW(t) using a CW = 5-year citation window (CW), meaning that only articles published within 5 years of p are included in the subgraph {c}_p = {c}_i ∪ {c}_j ∪ {c}_k.

Figure 2(a) illustrates the subgraph used to calculate CD_p for two hypothetical publications, a and b, which differ just in terms of {c}_k. Comparing the two subgraphs for a and b, the former schematic represents a paper from the 1980s when the median paper was cited just twice in its first 5 years from publication; whereas the latter schematic represents a paper published in the 2000s and accruing nine citations in the same amount of time. This difference alone corresponds to a 35% reduction in CD_b from CD_a. This reduction is characteristic of the average paper in MAG, as indicated by Figure 2(b), where the average CD₅(t) features a decline that is consistent with the overall trends reported in Figure 2 of Park et al. (2023). We note that the CD₅(t) curves reported in Figure 2 and Figures 6 and 9 of Park et al. (2023) are calculated according to broad disciplines, and show that disciplines with higher publication volumes—namely, life sciences and biomedicine, and physical sciences—naturally produce more references, and also tend to have smaller CD₅(t) values in any given year relative to the social sciences (e.g., JSTOR). A more definitive validation of this critique is provided by Macher et al. (2024), who found that many patent citations were omitted in the data set analyzed by Park et al. (2023). After accounting for the artificially reduced reference lists by incorporating the missing citations, Macher et al. (2024) find that CD₅(t) decreases in the early years to the extent that the negative trend reported by Park et al. (2023) disappeared entirely. Moreover, upon correcting for this systematic measurement bias attributable to the omitted patent citations, they instead find that the number of highly disruptive IT technology patents has actually increased since 1980. In a similar vein, Bentley et al. (2023) develop a weighted version of CD₅(t) to account for CI; after applying this modification to the scientific citation network, they report a significant positive trend in CD₅(t) starting in the 1990s and extending until the end of their data sample in the early 2010s. Moreover, Holst et al. (2024) recently show how the substantial subset of publications with zero references tend to be biased towards earlier years of scientific publishing and patenting; however, by construction according to the definition of CD, these items are assigned the maximum disruption value of CD = 1. Similarly, uncited papers also generate the modal disruption value of CD = 0, yet these papers and patents also tend to be biased towards earlier years (Kozlowski, Andersen, & Larivière, 2024). After correcting for these data quality artifacts and the systematic bias this incurs by excluding the publication and patent subsamples with CD = 1, Holst et al. (2024) show that temporal trends in CD become negligible for both patents and publications.

We also note that while the implementation of a CW may control for right-censoring bias, it does not control CI in any precise way. By way of example, consider the impact of the CW on N_k, the number of extraneous articles that do not cite p but do cite elements of {r}_p. A CW will reduce the number of papers contributing to CD₅(t) via N_k, but it will also reduce N_i + N_j in similar proportions, leaving the ratio R_k(t) unchanged, on average. Consider a more quantitative explanation that starts by positing that N_k increases proportional to n(t)r(t), as the nodes belonging to {c}_k are unconstrained by the first-order citation network {c}_i ∪ {c}_j ∪ {r}_p. As such, the term N_k is susceptible to CI according to two distinct channels, both r(t) and n(t). Following the same logic, N_i + N_j grows proportional to n(t). In both cases, even if the proportionality constant depends weakly on CW, the ratio R_k(t) will thus grow proportional to r(t).

There is likely to be considerable variance in the publication-level relationship between R_k,p and r_p, even comparing among publications with fixed r_p, because if any member of {r}_p is highly cited then N_k becomes skewed towards the heavy right tail of the citation distribution. This particular sensitivity of CD_p is illustrated Figure 2(a), which makes the case using median citation values, as representing the case for highly cited papers would be challenging to visualize accurately. This is because the base number of citations associated with extreme values in the citation distribution have increased dramatically over the last half century as a result of CI, such that the number of citations C(Q∣t) corresponding to the Q = 99th percentile of the citation distribution has grown at an annual rate of roughly 2%—increasing from roughly 55 citations in 1965 to roughly 125 citations in 2005—see Figure 4 in Pan et al. (2018). In this work we focus on the impact of CI manifesting through the growth of r(t), which has increased historically at roughly the same rate as C(99∣t), from roughly nine to 23 references per paper over the same period—see Figure 2(c). Consequently, R_k(t) ≫ 1 for nearly the entire period of analysis, where the growth of R_k(t) is largely explained by the growth of r(t)—see Figures 2(d) and 2(e). For this reason, it is more accurate to describe CD as converging to 0 as opposed to decreasing over time.

Figure 2(f) shows the marginal relationship between CD_p,5 and ln r_p, holding all covariates at their mean values. Results indicate a negative relationship between CD_p,5 and the number of references, consistent with our deductive argument. In particular, we measure a net shift of roughly −0.06 units in CD as r_p increases by a factor of 10 from five to 50 total references. As cross-validation, Leahey, Lee, and Funk (2023) also report a negative correlation between r_p and CD_p,5 resulting from a multivariate regression model for CD_p,5 including a broader panel of covariates (see Table 3 of that study, which focuses on disruption trends in sociology research).

Similarly, Figure 2(g) shows the positive correlation between CD_p,5 and ln k_p, corresponding to a shift of roughly +0.01 units as k_p increases by a factor of 10 from one to 10 coauthors. Despite this correlation representing a relatively small effect magnitude, it is nevertheless in stark contrast to the negative relationship reported by Wu et al. (2019). Interestingly, Leahey et al. (2023) also obtain a positive relationship between the number of coauthors and the disruption index calculated for both five and 10-year windows (see again Table 3 of that study). Considering the consistency between our result based upon large-scale data and their result based upon a narrow subfield, this would suggests that that descriptive results reported in Wu et al. (2019) are confounded by omitted variable bias. Moreover, the estimated coefficient for publication year in Leahey et al. (2023) is not statistically significant (i.e., it is indistinguishable from 0), which is inconsistent with the main result reported by the same authors in Park et al. (2023). Hence, as developed in the next section, these inconsistencies merit investigation by way of a mechanistic citation network model where confounding sources of variation can be fully controlled.

We employ computational modeling to explicitly control several fundamental sources of variation, and to also explore complementary mechanisms contributing to shifts in CD over time—namely, shifts in scholarly citation practice. Our identification strategy is to grow synthetic citation networks that are identical in growth trajectory and size, but differ just in the specification of (a) r(t) and/or (b) the rate of triadic closure denoted by β that controls the consolidation-disruption difference defining the numerator of CD.

We model the growth of a citation network using a model originally developed in Pan et al. (2018) that applies Monte Carlo (MC) simulation to operationalize stochastic link dynamics by way of a random number generator. This model belongs to the class of growth and redirection models (Barabàsi, 2016; Krapivsky & Redner, 2005), and reproduces a number of statistical regularities established for real citation networks—both structural (e.g., a log-normal citation distribution; Radicchi, Fortunato, & Castellano, 2008) and dynamical (e.g., increasing reference age with time (Pan et al., 2018); exponential citation life-cycle decay (Petersen, Fortunato et al., 2014)). The synthetic networks constructed and analyzed in what follows are available at the Dryad data repository and can be used to test CD and other citation-network based bibliometric measures for sensitivity to CI and other aspects of secular growth.

We construct each synthetic citation network by sequentially adding new layers of nodes of prescribed volume n(t) in each MC period t ≥ 0 representing a publication year. Each new node, denoted by the index a, represents a publication that could in principle cite any of the other existing nodes in the network. As such, the resulting synthetic networks are representative of a single scientific community, and also lack latent node-level variables identifying disciplines, authors, journals, topical breadth or depth, etc.

We seed the network with n(t = 0) ≡ 30 “primordial” nodes that are disconnected (i.e., they have reference lists of size r_a ≡ 0). By ensuring that the initial conditions are the same for all networks generated, in combination with the relatively large system size being evolved, we are confident that the long-term evolution of the system is sufficiently independent of the initial conditions and finite-size effects (i.e., the networks being simulated are uniformly sampled from the representative space).

To minimize variation in intracohort connectivity, all nodes added within a specific cohort t have reference lists of a common prescribed size, denoted by r(t). To model the exponential growth of scientific production, we prescribe the number of new “publications” according to the exponential trend n(t) = n(0) exp[g_nt]. We use g_n ≡ 0.033 as the publication growth rate empirically derived in prior work (Pan et al., 2018). Similarly, we prescribe the number r(t) of synchronous (outgoing) links per new publication according to a second exponential trend r(t) = r(0) exp[g_rt]. For both n(t) and r(t) we use their integer part, and plot their growth in Figure 3(a). We set the initial condition r(0) ≡ 25 in scenarios featuring no reference list growth (characterized by g_r = 0), such that each new publication cites 25 prior articles independent of t. Alternatively, in scenarios that do feature reference list CI, we use the empirical growth rate value, g_r ≡ 0.018 and r(0) ≡ 5. We then sequentially add cohorts of n(t) publications to the network over t = 1, …, T ≡ 150 periods according to the following link-attachment (citation) rules (network growth rules) that capture the salient features of scholarly citation practice:

1. System growth: In each period t, we introduce n(t) new publications, each citing r(t) other publications by way of a directed link. Hence, the total number of synchronous (backwards) citations produced in period t is C(t) = n(t)r(t), which grows exponentially at the rate g_C = g_n + g_r.

2. Link dynamics: Illustrated in the schematic Figure 3(b). For each new publication a ∈ n(t):

   • (i) 

     Direct citation a → b: Each new publication a starts by referencing one publication b from period t_b ≤ t_a (where t_a = t by definition). The publication b is selected proportional to its attractiveness, prescribed by the weight $P$_{b, t} ≡ (c_× + c_b,t)[n(t_b)]^α. The factor c_b,t is the total number of citations received by b through the end of period t − 1, thereby representing preferential attachment (PA) link dynamics (Baràbasi, Jeong et al., 2002; Jeong, Neda, & Barabàsi, 2003; Peterson, Presse, & Dill, 2010; Redner, 2005; Simon, 1955). The factor n(t_b) is the number of new publications introduced in cohort t_b, and represents crowding out of old literature by new literature, net of the citation network. The parameter c_× ≡ 6 is a citation offset controlling for the citation threshold, above which preferential attachment “turns on” (Petersen et al., 2014) such that a node becomes incrementally more attractive once c_b ≥ c_×.

   • (ii) 

     Redirection a → {r}_b: Immediately after step (i), the new publication a then cites a random number x of the publications cited in the references list {r}_b (of size r_b) of publication b. By definition, β represents the fraction of citations following from this redirection mechanism, which is responsible for the rate of nonspurious triadic closure in the network. Hence, by construction β = λ/(λ + 1) ∈ [0, 1], where λ represents the average number of citations to elements of {r}_b by publication a (such that the expected value of x is λ). Consequently, λ = β/(1 − β) is the frequency ratio for citations following the “redirection” mechanism (ii) to citations following the “direct” mechanism (i).

     We operationalize the stochastic probability of selecting x references according to the binomial distribution,

     $Px=k=rbkqk1−qrj−k,$

     (4)

     with success rate q = λ/r_b to ensure that 〈x〉 = λ. Put another way, on average, the total number of new citations per periods that follow from the redirection citation mechanism (ii) is r_(ii)(t) = βr(t).

     Once x is determined by way of a random number generator, we then select x_{Binomial(rb,q)} members from the set {r}_b (i.e., without replacement). Each publication belonging to {r}_b is selected according to the same weights $P$_p,t used in step (i). As such, this second-stage PA also prioritizes more recent elements of {r}_b (i.e., those items with larger t_p), in addition to more highly cited elements of {r}_b. Note that we do not allow a to cite any given element of {r}_b more than once within its reference list.

   • (iii) 

     Stop citing after reaching r(t): The referencing process alternates between mechanisms (i) and (ii) until publication a has cited exactly r(t) publications.

3. Repeat step 2 (Link dynamics) for each new publication entering in period t.

4. Update the PA weights, $P$_p,t, for all existing nodes at the end of each t.

5. Perform steps (1–4) for t = 1, …, T periods and then exit the network growth algorithm.

In this section we present the results of a generative citation network model (Pan et al., 2018) that incorporates latent features of secular growth and two complementary citation mechanisms illustrated in Figure 3(b), namely, (a) direct citation from a new publication a to publication b; and (b) redirected citations from a to a random number of publications from the reference list of b. The redirection mechanism (b) gives rise to triadic closure in the network, thereby capturing shifts in correlated citation practice—such as the increased ease with which scholars can follow a citation trail with the advent of web-based hyperlinks, as well as self-citation. This redirection is the dominant contributor to “consolidation,” which is measured by N_j in CD_p. We explicitly control the rate of (b) with a tunable parameter β ∈ [0, 1] that determines the fraction of links in the citation network resulting from mechanism (b). And to simulate the net effect of β, we construct some networks featuring a constant β(t) = 0 and other networks featuring an increasing β(t) ≡ t/400 such that β(t = 150) = 0.375 corresponding to roughly 1/3 of links arising from mechanism (b) by the end of the simulation—see Figure 3(c).

We construct ensembles of synthetic networks according to six growth scenarios that incrementally add or terminate either of two citation mechanisms: g_r = 0 corresponds to no CI; and β = 0 corresponds to no triadic closure (i.e., no “consolidation”). More specifically, the parameters distinguishing the six scenarios analyzed in what follows are

1. No CI (g_r = 0 with r(t) = 25); and no explicit redirection mechanism that controls triadic closure (β = 0);

2. No CI (g_r = 0 with r(t) = 25); and an increasing redirection rate, β(t) = t/400 such that β(150) = 0.375;

3. CI implemented using the empirical value (g_r = 0.018) with r(0) = 5; and increasing redirection rate, β(t) = t/400;

4. Same as (3) but calculated using a larger citation window;

5. Same as (3) but reference list capped at r(t) = 25 for t ≥ T* ≡ 92; and

6. Same as (4) but reference list capped at r(t) = 25 for t ≥ T*.

Figure 3(d) shows 16 average CD(t) curves calculated for each synthetic network. Because the sources of network variation are strictly limited to the stochastic link dynamics, there is relatively little variance across each ensemble of networks constructed using the same scenario parameters, so in what follows we show all realizations simultaneously. As there are no latent institution, author or other innovation covariates, the difference between network ensembles is attributable to either CI or the redirection mechanism.

We start by considering scenarios (1, 2) for which g_r = 0, which show that CD₅(t) systematically increases in the absence of reference list CI. While scenario (1) does capture CI attributable to increased publication volume (g_n > 0), it does not appear to be sufficient to induce a negative trend in CD₅(t). Scenario (2) features an increasing β(t), which results in larger CD values because redirected citations tend to fall outside shorter CW and thus are not incorporated into the CD subgraph. Summarily, comparison of (1) and (2) indicates that the redirection mechanism capturing shifting patterns of scholarly citation behavior is the weaker of the two mechanisms we analyzed.

The comparison of scenarios (2, 3) illustrates the role of CI. Notably, scenario (3) reproduces both the magnitude and rate of the decreasing trend in CD(t) observed for real citation networks (Park et al., 2023). Figure 3(e) shows an alternative metric $CD5nok$ proposed by Bornmann et al. (2020), which also matches the empirical trends reported by Park et al. (2023). These results demonstrate the acute effect of reference list CI on CD, as the only difference between scenarios (2) and (3) pertains to g_r.

Figure 3(f) reproduces the linear relationship between r(t) and R_k(t) and confirms the empirical relationship shown in Figure 2(e)—thereby solving the mystery regarding the origins of the decreasing disruptiveness over time (Kozlov, 2023): As the size of the reference list {r}_p increases, so does the likelihood that {r}_p contains a highly cited paper, which increases N_k to such a degree that R_k,p ≫ 1 and so CD_p → 0, independent of the relative differences between disruption and consolidation captured by N_i − N_j. Figure 3(g) shows that even $CD5nok$ suffers from systematic bias affecting its denominator, so neglecting the term N_k does not solve the fundamental issue of CI.

Scenarios (3, 4) reveal the effect of CW, which controls the size of the set {c}_p and thus the magnitude and growth rate of R_k(t). Notably, the number of items included in {c}_p depends on both CW and t because the reference age between the cited and citing article increases with time (Pan et al., 2018). Regardless, the average CD_CW(t) → 0 as r(t) increases, independent of CW.

Figure 4 further explores the implications of CI on CD by modeling a hypothetical scenario in which CI is suddenly “turned off” after a particular intervention time period T*. In this way, scenarios (5, 6) explore the implications of a restrictive publishing policy whereby all journals suddenly agree to impose caps on reference list lengths. Scenarios (5, 6) enforce this hypothetical policy at t ≥ T* ≡ 92 by way of a piecewise smooth r(t) curve such that: r(t) = r(0) exp[g_rt] for t < T* and r(t) = r(T*) = 25—see Figure 4(a). This hypothetical intervention exhibits the potential for the scientific community to temper the effects of CI by way of strategic publishing policy. For completeness, scenarios (3) and (5) use CW = 5 and scenarios (4) and (6) use CW = 10.

Figures 4(b, c) verify that the average CD(t) and R_k(t) trajectories for each pair of scenarios are indistinguishable prior to T*. Yet immediately after T*, scenarios (5) and (6) diverge from (3) and (4), respectively. Notably, the average CD₅(t) in scenarios (5) and (6) reverse to the point of slowly increasing, thereby matching the trends observed for scenario (2). In the spirit of completeness, Figure 4(c) confirms that this trend-reversal is due to the relationship between r(t) and R_k(t). The shifts in the average CD₅(t) are indeed representative of the entire distribution of CD_p,5 values—see Figures 4(d, e). Interestingly, the distribution P_t(CD₅) converges to a stable Extreme-Value (Fisher-Tippett) distribution in the absence of reference list growth, which exposes candidate avenues for developing time-invariant measures of disruption by rescaling values according to the location and scale parameters. The feasibility of this approach was previously demonstrated in an effort to develop field-normalized (Radicchi et al., 2008) and time-invariant (z-score) citation metrics (Petersen, Ahmed, & Pavlidis, 2021; Petersen, Majeti et al., 2018a).

In general, the benchmark for bibliometric analysis, and more generally in econometric analysis, is establishing that the measurement framework is not biased by confounders or other forms of systematic bias, so that results reflect phenomenological trends and not methodological bias. In practice, this is traditionally achieved by way of demonstrating that the metrics being used are stationary over time, such that the first moments (mean and standard deviation) of the metric distribution are time-independent. In multivariate regression methodologies, this can be approximated by including temporal dummy variables (fixed effects) that control for time variation in the first moment of the dependent variable. To promote a better understanding of the implications of citation inflation in temporal network analysis, we have provided the full ensemble of synthetic networks analyzed here, which can be used to test alternative citation-based indices for temporal systematic biases emerging from the growth dynamics of the network—see the Data availability statement for the weblink.

In summary, despite the reasonable logic behind the definition of CD, we show that its numerator (which captures the difference between disruptive and consolidating links, N_i − N_j) is systematically susceptible to becoming overwhelmed by the extensive quantity R_k ∼ r(t) appearing in the denominator. We further contextualized this deductive critique by showing that the CD index artificially decreases over time due to citation inflation deriving from ever-increasing r(t), rendering CD systematically biased and unsuitable for cross-temporal analysis. This conclusion is further supported by two recent methodological critiques of (Park et al., 2023), one focusing on patents (Macher et al., 2024) and the other on scientific publications (Bentley et al., 2023). The former corrects for omitted patent citations, whereas the latter defines a weighted variant of CD₅(t) that more appropriately accounts for CI. Following their proposed modifications, both studies report an increasing rate of disruption in recent decades, likely owing to the techno-informatic revolution that has transformed the scientific endeavor—and is manifest in bibliometric studies using quantities and model designs that are resistant to CI (Petersen, 2022; Petersen et al. 2018a, 2021; Yang, Pavlidis, & Petersen, 2023).

For the same reasons that central banks must design monetary policy to avoid the ill effects of printing excess money (Orphanides, 2003; Orphanides & Solow, 1990), researchers analyzing scientific trends should be wary of citation-network bibliometrics that are not stable with respect to time. Scenarios where achievement metrics are nonstationary and thus systematically biased by nominal inflation are common, including researcher evaluation (Petersen et al., 2010), journal impact factors (Althouse, West et al., 2009), and even achievement metrics in professional sports (Petersen & Penner, 2020; Petersen, Penner, & Stanley, 2011). Yet our results do not point to a straightforward way to address this methodological issue with the existing formulation of the disruption index. First, as illustrated in Figures 1(a) and 2(a), the impact of exponentially growing r(t) and n(t) is nonlinearly incorporated into the definition of CD, so there is no easy way to address this bias at the metric level. Second, while Figure 4 suggests that there is a stable distribution of CD, these results are conditioned on publication cohorts with fixed reference list lengths. Yet in general, publications from even the same year feature a broad frequency distribution P(r_p,t), which explains why the unconditional CD distribution is asymmetric and highly skewed, and far from normal—see Wu et al. (2019). Hence, distribution-level normalization (also referred to as detrending and standardization) is not likely to be straightforward, because the Extreme Value (Fisher-Tippett) distribution that emerges in Figure 4 belongs to the double-exponential class of distribution functions.

In addition to the measurement error induced by CI, the disruption index also does not account for confounding shifts in scholarly citation practice. The counterbalance to disruption, captured by the term N_i in Eq. 1, is consolidation (N_j), which is fundamentally a measure of triadic closure in the subgraph G_p. While triangles may spuriously occur in a random network, their frequency in real networks is well in excess of random base rates due to the correlated phenomena underlying the scholastic practice—in particular, the strategic (personal and social) character of scholarly citing behavior that is increasingly preconditioned by implicit and explicit bias embedded within the digital platforms that mediate the search and recommendation of scientific literature (Helbing & Abeillon, 2023).

The source and implications of citation inflation are not inherently undesirable, and if anything point to thriving industry emerging from the scientific endeavor. The advent of online-only journals is a main reason for the steady increase in r(t), as they are not limited by volume print capacity, unlike more traditional print journals. In the present era of megajournals (Petersen, 2019), there may be a tendency to cite more liberally than in the past. Another mechanism connecting CI and citation behavior derives from the academic profession becoming increasingly dominated by quantitative evaluation, which thereby promotes the inclusion of strategic references dispersed among the core set of references directly supporting the research background and findings (Abramo, D’Angelo, & Grilli, 2021). Notably, scholars have identified various classes of self-citation (Ioannidis, 2015), which generally emerge in order to benefit either the authors (Fowler & Aksnes, 2007; Ioannidis, Baas et al., 2019; Pinheiro, Durning, & Campbell, 2022), institutional collectives (Qiu, Steinwender, & Azoulay, 2024; Tang, Shapira, & Youtie, 2015), the handling editor (Petersen, 2019), and/or the journal (Ioannidis & Thombs, 2019; Martin, 2016)—but are otherwise difficult to differentiate from “normal” citations. Regardless of their intent, these self-citations are more likely to contribute to triadic closure because if article b cites c as a result of self-citation, then for the same reason a new article a that cites b (or c) is that much more likely to complete the triangle on principle alone.

These two issues—citation inflation and shifting scholarly behavior—introduce systematic bias in citation-based research evaluation that extends over significant periods. Indeed, time is a fundamental confounder, so to address this statistical challenge various methods introducing time-invariant citation metrics have been developed (Petersen et al., 2014, 2018a, 2018b, 2021; Radicchi et al., 2008). A broader issue occurs when different variables simultaneously shift over time, such as the number of coauthors, topical breadth, and depth of individual articles, which makes establishing causal channels between any two variables ever more challenging. By way of example, we analyzed the relationship between CD_p and k_p, using a regression model with fixed effects for publication year to superficially control for secular growth, and observe a positive relationship between these two quantities, in stark contrast to the negative relationship reported by Wu et al. (2019).

We conclude with a policy insight emerging from our analysis regarding interventional approaches to addressing citation inflation. Namely, journals might consider capping reference lists commensurate with the different types of articles they publish (e.g., letters, articles, reviews). An alternative that is more flexible would be to impose a soft cap based upon the average number of references per article page (Abt & Garfield, 2002). The results of our computational simulations indicate that such a policy could readily temper the effects of citation inflation in research evaluation, and might simultaneously address other shortcomings associated with self-citations by effectively increasing their cost.

We thank the anonymous reviewers for their timely and astute comments that helped us to improve the report.

Alexander M. Petersen: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Visualization, Writing—original draft, Writing—review & editing. Felber Arroyave: Conceptualization, Writing—review & editing. Fabio Pammolli: Conceptualization, Methodology, Supervision, Writing—review & editing.

The authors have no competing interests.

AMP acknowledges financial support from a Hellman Fellow award that was critical to completing this project.

All synthetic citation networks analyzed are openly available at the Dryad data repository: https://doi.org/10.6071/M3G674. The pseudocode for the citation network growth is sufficient to generate additional citation networks with different parameters.