Categorization of scientometric data in a Benfordian context Open Access

https://www.webofscience.com/api/gateway/wos/peer-review/10.1162/qss_a_00361

This law is surprising, as one would expect a uniform distribution of the different digits. Numerous publications have given different explanations for this unexpected regularity: Gauvrit and Delahaye (2008) recall the well-known fact (cf. for example, Diaconis, 1977) that the uniformity of the logarithmic mantissa log X of a strictly positive real random variable X is equivalent to Benford’s law on X. They propose a sufficient condition for a random distribution to generate numbers that satisfy this law: If a statistical distribution concentrates a significant number of low and high values and has a certain regularity, it roughly conforms to this law.

The work of Delahaye (2012) and Kafri (2023) shows similarities between Zipf’s law and Benford’s law. The latter shows that under certain conditions (Eq. 2) is obtained by applying the Riemann sum to Zipf’s law (Kafri, 2023).

In 1976, Raimi wrote an article on Benford’s law called The First Digit Problem, in which he presented a first state of the art of the work (Raimi, 1976, p. 521). He presents 37 papers in chronological order with the aim of retaining only the papers: “deliberately omitting only those references to the problem which make no attempt to add to its understanding … among authors of which are mathematicians, statisticians, economists, engineers, physicists and amateurs.” He also proposed 15 additional bibliographical references which “[do] not refer explicitly to the problem.”

In his book, Nigrini returned to the works identified around Benford’s law (Nigrini, 2012, p. 293). In 2000, he identified 200 articles on Benford’s law. Eleven years later, he had identified 750 papers on the subject. He continued the reasoning of Raimi (1976) by suggesting a difference between statistics which deals with real data and mathematics. He pointed out that Benford’s law is interested in both aspects. That it motivates the studies of both of these fields. He proposed a classification of the works in eight points with three main categories: papers that prove Benford’s law; those that have an approach to mathematical phenomena; and those that consider Benford’s law as a test, particularly in terms of fraud.

The applicability of Benford’s law is very interesting. This law is verified in many collections of numbers enumerating objects of various origins: the number of inhabitants of cities, the distances between stars, the lengths of rivers, the prices in supermarkets, and the citations of journals in a database. Not only does Benford’s law cover many areas, but it is regularly used and cited in different fields. A quick study of the Web of Science (WoS) among the 50 most cited articles dealing with Benford’s law shows that the fields concerned are, of course, fraud (whether in the financial or electoral field (Mebane, 2011; Tam Cho & Gaines, 2007), but also on electrical networks (Wei, Sundararajan et al., 2017); scientific data (Diekmann, 2007; Geyer & Williamson, 2004; Judge & Schechter, 2009); astronomy (Alexopoulos & Leontsinis, 2014); atomic physics (Pain, 2008) and quantum physics (Rane, Mishra et al., 2014); hydrology (Nigrini & Miller, 2007); image processing (Fu, Shi, & Su, 2007; Perez-Gonzalez, Heileman, & Abdallah, 2007); natural sciences (Sambridge, Tkalčić, & Jackson, 2010); and, more recently, in epidemiology with COVID-19 (Lee, Han, & Jeong, 2020).

Scientific integrity and trust are the cornerstones of scientific research, because of their relationship with constantly evolving knowledge. In recent years, Benford’s law has become a tool for detecting scientific fraud (Eckhartt & Ruxton, 2023). Reproducibility in academic research has long been a persistent problem, in contradiction with one of the fundamental principles of science. The growing number of false claims found in academic manuscripts is worrying. This goes against the very nature of science and calls into question the reproducibility of academic research. Lazebnik and Gorlitsky (2023)’s work has determined the rate of manipulation in academic research. Furthermore, Schumm, Crawford et al. (2023) offer an approach designed to detect issues in social science surveys when dealing with small sample sizes. An application of this law to medical data has been proposed by Hein, Schuepfer, and Konrad (2011). More recently, the work of Gupta, Singh, and Banshal (2023) proposes that Benford’s law can be used to define a framework for assessing the quality of altmetric data.

There are fewer studies in the field of scientometrics, but the theoretical developments and experiments are just as relevant. Over the last decade, several studies have been carried out on data from WoS and Scopus. One focuses on the number of articles, citations, and impact factor of WoS from 1998 to 2007 (Campanario & Coslado, 2011). A second study aims to compare this law on data from WoS and Scopus (Alves, Yanasse, & Soma, 2014).

Benford’s law is generalized for the other digits. The work of Alves, Yanasse, and Soma (2016). focuses on Benford’s law for the second digit.

The work of Hürlimann (2015b) discussed parametric extensions of Benford’s law. For this purpose, they used the mean absolute deviation (MAD), test developed by Nigrini (2012) for their experiment. All of these studies tend to show that the data in these databases, which are at the source naturally produced by researchers, agree in verifying this law.

While Benford’s law is well known and studied, there remains a question mark over the nature of the data studied in the field of scientometrics. Before experimenting with Benford’s law on bibliometric data for the purposes of fraud detection or the regulation of science, it is necessary to understand the nature of these indicators in a Benfordian context. The research question concerns the nature of scientometric data.

Indeed, we have found that previous studies of Benford’s law on scientometric data have not explained its behavior in relation to the nature of the scientometric information used. On page 431 of Campanario’s article, the authors tested the distribution on articles, citations and impact factor (Campanario & Coslado, 2011). Their conclusion is as follows: “We have no explanation for these differences.”

Egghe’s work based on Campanario’s data stated in the conclusion that “We consider this to be an interesting discovery, but we have no informetric explanation for it” (Egghe & Guns, 2012, pp. 1663, 1665). Alves’ work, published in 2014 following Campanario and Egghe, did not offer a discussion of the nature of the objects studied, but wondered about anomalies in the data sets and the ability to detect them (Alves et al., 2014). Despite these previous works, there is still a lack of understanding when it comes to reading Benfordian distributions.

Therefore, we proposed a typology of the scientometric objects studied, based on the different approaches to a Benfordian distribution. Consequently, in this article we have constructed a new data set extracted from WoS from 1997 to 2019 and from Scopus from 1999 to 2019. We have also taken into account data from the work of Campanario and Coslado (2011) and Alves et al. (2014). This led us to a corpus of 181 distributions from different scientometric objects. The data set we used consists of unaggregated data. We have made available all the distributions analyzed. They are both included in the Supplementary material and available for download via Zenodo (Bertin & Lafouge, 2024).

The article is structured as follows: After recalling Benford’s law and its generalized form in Section 2, Section 3 describes the construction of the scientometrics data set from WoS, Scopus, and historical data.

Next, we present the tests used in this work. Two well-known tests were employed: the MAD test, developed specifically for this law, and the classic chi-square test. Furthermore, a third test, called Max, is proposed, which represents a compromise between the macro and micro tests.

Section 4 presents the results obtained and a discussion that proposes an analysis at the micro and macro levels. The results lead us to categorize the scientometric data in our study and to propose a typology of distributions based on the results. The distinction between macro and micro data and the tests used allow us to hypothesize about the classes that define the nature of the scientometric information used. It allows us to provide some answers about the nature of the scientometric information used and to better understand the Benfordian phenomena in a scientometric perspective.

Section 5 concludes the study with a summary of this work and of the main results obtained.

In this formalism Benford’s law can be stated as follows: A digit d = 1, …, 9 (sources) produces the numbers (items) whose first digit is d. This scheme is the same in Zipf’s law. In this case the number of sources is finite. This scheme will allow us to modify how the law is written and, consequently, to offer a proposal for generalization.

This form led Egghe to generalize this law.

The limit of the generalized law when β → 1 is thus equal to the original Benford’s law (Eq. 1). This unproven result in Egghe’s article reinforces this generalization in our view.

This generalized law can be perceived through the scale invariance of power laws (Raimi, 1976, p. 529; Pietronero, Tosatti et al., 2001).

The construction of the data set is based both on historical data, for the purposes of validation and reproducibility of this experiment, and also on WoS and Scopus data in order to have a set of scientometric objects such as the impact factor, h-index, number of journals, references, citations, and articles over different periods. In order to respond to the problem posed, namely the understanding of bibliometric objects in a Benfordian context, we will observe the behavior of these scientometric objects during the application of the tests. The latter invites us to reflect on the nature of scientometric objects through discussion.

For the construction of the data set that will be used for our experimentation, we retained the data set used by Alves et al. (2014). This is a compilation of the Campanario and Coslado (2011) and Alves et al. (2014) data sets used by Hürlimann (2015a) and Egghe and Guns (2012). The data set produced, contains the historical data but also the new data set we produced from WoS and Scopus.

To create the new data set to test Benford’s law, we collected for each WoS journal between 1997 and 2019 the impact factor and the total number of citations. For each Scopus journal between 1999 and 2019, we collected the h-index, the number of cumulative citations over three years, the number of references over the year, and the number of articles over the year. We also collected the ratio of the number of references to the number of articles. The ratio is a bibliometric indicator, as are the h-index and the impact factor.

For these 181 distributions of WoS and Scopus, we calculate the percentage of the first digit. The results are shown in Tables S1–S10 in the Supplementary material. It was necessary to change the scale to process the 23 distributions of impact factor: We multiply all these values by 10,000. For example, the Biologia journal whose impact factor was 0.159 in 2000 will now has a value of 1,590. Benford’s law has been tested on data extracted from the WoS and Scopus databases.

The optimal β is then chosen when D_β is minimal with an error range of 0.01.

The values are expressed in percentages in all the tables. To measure the adequacy of the two theoretical distributions with the 181 observed distributions we use three tests.

Equation 13 is only valid for the Benford’s law. It shows us the dependence on the number of journals: If two observed distributions are identical, one for 10,000 journals and the other for 20,000 journals, the χ² will necessarily be in a ratio of 2.

This dependence of χ² on the number of items (here journals) leads researchers to develop other tests.

We then carry out the classical test of the adjustment of χ²: We will carry out this test with a precision of 5%. The threshold of rejection read in the table for eight degrees of freedom (Benford’s law) is then 15.51 and for seven degrees (generalized Benford’s law) it is 14.07.

The MAD is not a classical test as it does not depend on the size of the sample. It assumes that we know the FSD perfectly. Here, it is considered as a conformity indicator. It is also possible to use the mean χ² (weighted least squares (WLS): chi-square divided by sample size). This measure is more of an empirical rule in order to choose the best adjustment (Hürilmann, 2015b, p. 355). We should also mention the recent application (Cerqueti & Lupi, 2022) of another type of test for large samples: the severity principle.

Table 1 was provided by Nigrini to clarify conformity with the MAD (Nigrini, 2012, p. 160). We adopt these notations in all tables.

It is relevant at this step of the discussion to add a complementary test denoted Max for a better interpretation of the data.

This test constructed here is of the same nature as the one using the Z-statistic (Alves et al., 2016, p. 1492).

We note Δ_p = t_α · $f1−fN$⁠.

We then carry out a goodness-of-fit test on Max with a precision of 5%. The calculation of Fmax is done in Tables 3–7.

In summary, these three tests are quite different and seem necessary to validate the adjustments. The MAD test, specifically designed for Benford’s law, is unavoidable due to its widespread use in many studies. However, it is not a classical test and of course it is inoperative for the generalized Benford’s law. The chi-square test is the statistical test that seems to us the most relevant for this type of discrete law. However, as this law does not depend on any parameter, we have seen that it is very sensitive to the size of the population. Therefore, it seemed necessary to construct a third test. This does of course depend on the size of the population, but its rejection threshold is linked to the root of the population and is therefore less sensitive.

This section is in three parts. The first part, microanalysis, focuses on distributions over time (i.e., the calculations are made year by year).

Our data is presented in Tables 3–6. Additionally, the results from historical data are presented in Table S7.

The second part proposes a macroanalysis, aggregating distribution data over several years, leading us to construct Table 2. This part also takes into account historical data. The final section presents a categorical organization of the scientometric data correlated with the tests used in this study. The results obtained, which will be discussed, are shown in Table 2.

In this study, we therefore examined 181 distributions. The distribution of the data is as follows. WoS provides 23 distributions for citations and 23 distributions for impact factor. The historical data set provides 10 distributions for citations, 10 distributions for articles and 10 for the impact factor. Finally, Scopus provides 21 distributions for citations, 21 for the h-index, 21 for the number of articles, 21 for the number of bibliographic references and 21 for the ratio. The 181 distributions analyzed can be found in Supplementary material: Tables S1–S10.

The three previous tests are implemented. In all cases, as expected, we observe a hierarchy of validity between the three tests: If the χ² test validates a fit, then the Max test also validates it, and when it makes sense, the MAD test is also validated, with “Close Conformity” or “Acceptable Conformity” as the critical value. We test the validity of the generalized Benford’s law in all cases. Remember that the studies cited concerning historical data only use the χ² test (Campanario & Coslado, 2011; Egghe & Guns, 2012).

The results are in Tables 3–6. For each year, we tested the validity of Benford’s law and the generalized Benford’s law. For these tables the results are read as follows:

• Column 1: year

• Column 2: number of journals analyzed (the number of journals analyzed may be different for two indicators in the same year)

• Column 3: MAD critical value (see Table 1)

• Columns 4, 5, and 6: calculation of the three indicators Fmax (Eq. 17), Max (Eq. 15) and distance of χ² (Eq. 12).

The green shaded boxes are those where Benford’s law is validated by χ² for a 95% confidence level. The red shaded and bold boxes are those where Benford’s law is validated by the Max test for a 95% confidence level. We then test the generalized Benford’s law.

• Column 7: value of β. When the column is empty, it means that the generalized Benford’s law does not improve the original: In this case, the optimal β is 1

• Columns 8 and 9: calculation of Max and χ² as before

The purple shaded and underlined boxes are those where the generalized Benford’s law is validated by χ² for a 95% confidence level. The red shaded and bold boxes are those where the generalized Benford’s law is validated by the Max test for a 95% confidence level.

The results can be summarized as follows. For the 46 WoS distributions, we have

1. The total number of journal citations (Table 3): The critical value of the MAD is always “Close conformity”; the Max test agrees in all cases. The χ² test validates 82% of cases.

   For the generalized Benford’s law, the result is the same.

2. The impact factor (Table 3): The critical value of the MAD is “Close Conformity” for only nine cases and “Non Conformity” for three cases; 39% of cases are validated by the Max test. Only 13% of cases are validated by the χ² test.

   For the generalized Benford’s law, the result for the χ² test is the same; for the Max test the result is better: it validates 80% of cases. β varies between 0.97 and 1.25.

1. The number of cumulative citations over three years (Table 4): The critical value of the MAD is always “Close Conformity.” The Max test agrees in all cases. Only one is validated by the χ² test.

   For the generalized Benford’s law, the χ² test validates the law in 80% of cases and in 100% of cases for the Max test. β varies between 1.02 and 1.08.

2. h-index (Table 5): The critical value of the MAD is “Close Conformity” for only nine cases and “Acceptable Conformity” for the other cases. The Max test validates only one case. The χ² test is not valid in any cases.

   For the generalized Benford’s law, the Max test validates four cases. The χ² test is not valid in any cases. β varies between 0.90 and 0.97.

3. Number of bibliographic references/Number of articles (ratio) (Table 5): The MAD critical value is “Non Conformity.” The Max and χ² tests are never validated. The generalized Benford’s law introduces no modification.

4. Number of bibliographic references (Table 6): The MAD critical value is “Close Conformity” for 1 case and “Acceptable Conformity” for 20 cases. The Max and χ² tests are never validated. The generalized Benford’s law introduces no modifications.

5. Number of articles published (Table 6): The MAD critical value is “Acceptable Conformity”. The Max and χ² tests are never validated. The generalized Benford’s law is validated in only one case by the Max test.

The results obtained are presented in Table 7. We apply our analysis tools to the 2011 scientometric data (Campanario & Coslado, 2011) on the number of articles, the number of citations and the impact Factor from 1998 to 2007 of WoS journals. The results are gathered in Table 7. They follow the same presentation as our results in Table 3. The test used by the authors is χ². Obviously, we obtained the same result as the authors on this test with their data. There is a difference between our calculation and that of the authors because we do not analyze the same number of journals. However, we note that when the value is different, the ranking is very close. We proceed with the MAD test, the Max test, and an adjustment with the generalized Benford’s law. The results for the citations and impact factor data set between 1998 and 2007 are compared with the result obtained from our own collection of WoS data. We were not able to collect the data set corresponding to the number of articles.

1. Citations (Table 7): The result is expected (see Table 3). The MAD critical value is “Close Conformity” and the Max test is valid in all cases. The χ² test is valid in all cases.

2. Number of articles (Table 7): The MAD critical value is “Close Conformity” in one case and “Acceptable Conformity” in the other cases. The χ² test is never significant. The Max test is significant in eight out of 10 cases. For the generalized law, the Max test is always validated and the χ² test is significant in six cases. β varies between 0.87 and 0.92.

3. Impact factor (Table 7): If we compare the results with those of Table 3, the MAD and Max tests are almost identical. The χ² test validates three cases, one more than for our data. The Max test validates all cases, as is the case with our data set. For the generalized law, the χ² test is significant in five cases. The β test ranges from 0.97 to 1.1.

In summary, the results are almost identical for both data sets. The differences are due to the different number of journals analyzed. For example, for the year 2003 and the impact factor 7,548 journals are collected in one case and 5,845 in the other.

After an initial study of the years, we propose a macroanalysis of the 181 distributions by aggregating the temporal data. A digit-by-digit analysis (i.e., a column-by-column analysis of the 90 distributions in the appended tables) is carried out. We calculate the mean, median, and standard deviation of the nine distributions of digit values over the total number of years (23 for Scopus, 21 for WoS, 10 for historical data). We will define the Benfordian distribution of averages.

This distribution is all the more relevant if the mean is a representative of the value of each digit for all periods. A quick examination of the tables in the appendices where mean, median, and standard deviation are calculated shows that the distribution of values has a Gaussian appearance (median and mean close together) with a low standard deviation. We have represented the histogram of the distribution of the 23 values of digit 1 of the WoS citations which have an average of 29.93 (see Figure 2). This leads us to calculate the average distributions for each data set.

In order to produce Table S7 in Supplementary material, which allows us to analyze, we then use these 90 distributions to construct the 10 average distributions. Each digit average is compared with the corresponding theoretical value. Except for the data set corresponding to the ratio Number of references/Number of articles (see Table S7), good conformity with the theoretical value is observed.

Only the MAD test is possible, as the number of journals varies from year to year. Except for the ratio data set, the other distributions are C or AC. The three citation distributions are of type C.

This work has made it possible to apply Benford’s law and the various related tests to several bibliometric objects. By considering categorizations, macro- and microanalyses have laid the foundations for a reflection on the nature of the scientometric objects used. Macroanalysis shows the nine distributions of average have MAD values of C or AC (see Table 2). Note that the distribution of ratios has not been taken into account.

In the microanalysis, Benford’s law was first tested with the three tests MAD, χ², and Max. From a quantitative and global point of view, the results obtained for the 160 distributions show

• 154 distributions (96%) are of type C or AC;

• 58 distributions (36%) validate the χ² test for the generalized Benford’s law; and

• 98 distributions (61%) validate the Max test for the generalized Benford’s law.

However, microanalysis allows us to refine the results and to observe differences between the different types of distributions. We then grouped the homogeneous distributions into three classes (see Table 2);

• A first class: the 54 distributions of citations produced by scientific activity;

• A second class: the 52 distributions relating to the number of references and the number of articles produced; and

• Finally, a last class: the 54 distributions concerning the scientometric indicators h-index and impact factor, which are metrics of science.

They clearly show that we can classify the three categories according to the success of the statistical tests (χ² and Max) (see Table 2).

First, we get citations (100% Max, 56% χ²), second, indicators (35% Max, 11% χ²), and third, references and articles (17% Max, 0% χ²).

Second, we tested the generalized Benford’s law. The ranking remains unchanged. In all three categories, the generalized law improves the results. The range of variation of β is only 0.11. The most significant improvement is for items in the historical data set (Table 7).

How can we explain these differences in applicability? We have observed Benford’s law on a wide range of scientometric data. There are scientometric distributions that do not verify Benford’s law, and these phenomena invite us to reflect on their nature. We can divide the scientometric objects into two main categories, namely Product of Science and Metrics of Science. The Product of Science class contains citations, references, and articles. The Metrics of Science class contains impact factor, h-index, and ratio. To understand this phenomenon, we’ll discuss the following: citations, references, and articles, which belong to the Product of Science class, and bibliometric indicators, which belong to the Metrics of Science class.

Benford’s law applies particularly well to bibliographic citations.

Whatever the data set considered, the citation distributions follow Benford’s law (Gauvrit & Delahaye, 2008). Indeed, citations are data that reflect the actual use of scientific articles by other researchers. Scientometrics has shown that, very often, only a few articles receive a large number of citations, while the majority of articles receive far fewer. They often follow an uneven distribution, with a few articles receiving a large number of citations, while the majority of articles receive fewer.

The number of references in an article, unlike the number of citations, is limited by the researcher’s practice: There are rarely extremes. For the number of articles in a journal, the constraint is imposed by the edition itself. Thus, for this third class of scientometric elements, the results obtained show poor results for the application of Benford’s law. Although this may seem paradoxical at first glance, because it’s easy to replace the term citation with references, it’s important to understand that citations are generated naturally.

References and the number of articles, on the other hand, are limited by editorial practices. Indeed, the length of an article is artificial, in the sense that the length of the article is limited. This also implies a constraint on bibliographic references. As the length of articles is limited, the use of references is also limited. These constraints also affect the distribution of significant digits, which partly explains the poor results obtained.

When considering bibliometric indicators, their applicability depends on the specific characteristics of the bibliometric data in question.

The impact factor is a specific indicator that measures the frequency with which a journal’s articles are cited in other articles over a given period. It is calculated as the average number of citations received by a journal’s articles over a given period (e.g., 2 years). The h-index reflects both the number of publications and the number of citations of a researcher. The ratio does not follow Benford’s law. One possible interpretation is that references and the number of articles are limited by editorial practice. As the length of articles is limited, so is the use of references. However, we have not categorized the journals, whether paper, digital, or megajournal, and this work remains to be done.

The generalized Benford’s law naturally gives better results when fitting a distribution. Indeed, Section 2 shows that its generalization consists in adding a parameter, as for Zipf’s law, which in its primitive form has no parameter. This improvement can be seen when we compare the indicator values of χ² and Max for the two laws (see Tables 3–6). We will also observe that the generalized Benford’s law is all the more relevant as the scientometric objects that constitute the three classes respect Benford’s law (see Table 2).

In our experiment, the generalized Benford’s law does not provide any results for distributions that are not Benfordian. It is important to point out that Benford’s law is not universal and does not apply to all types of scientometric data; see for example the distribution of ratio. Indeed, distributions that are not constrained by the system or by humans better verify Benford’s law. This partly explains the open questions in the literature (Alves et al., 2014; Campanario & Coslado, 2011; Egghe & Guns, 2012).

This paper focuses on scientometric objects and their behavior. It is important to consider the nature of scientometric observables. To this end, we have built a corpus that allows us to experiment with different tests. All the data produced are freely available for reproducibility purposes. We have also confirmed the results of our colleagues. We used MAD, Max, and χ², applying Benford’s law and the generalized Benford’s law. The latter confirms its generalization for certain scientometric objects, such as citations.

We have shown that Benford’s law applies particularly well to citations and, to a lesser extent, to bibliometric indicators. Benford’s law is not easily applicable to bibliographic references and articles. After proposing a categorization, we put forward an explanation of this phenomenon, together with the constraints that apply to these objects.

The next steps in this work are to extend the scientometric objects. Indeed, we can consider altmetrics, as suggested by the work of Gupta et al. (2023). This work is based on an abductive approach based on observations of data sets, so it would be relevant to mathematize the concept of constraint around scientometric objects.

The authors would like to express their gratitude to the reviewers for their valuable feedback, which has been incorporated into this version of the manuscript.

Marc Bertin: Conceptualization, Data curation, Funding acquisition, Investigation, Project administration, Resources, Supervision, Validation, Visualization, Writing—original draft, Writing—review & editing. Thierry Lafouge: Conceptualization, Formal analysis, Investigation, Methodology, Software, Supervision, Validation, Visualization, Writing—original draft, Writing—review & editing.

The authors have no competing interests.

The data used in this study are published in https://doi.org/10.5281/zenodo.12698510.