Abstract

The most commonly used publication metrics for individual researchers are the total number of publications, the total number of citations, and Hirsch’s h-index. Each of these is cumulative, and hence increases throughout a researcher’s career, making it less suitable for evaluation of junior researchers or assessing recent impact. Most other author-level measures in the literature share this cumulative property. By contrast, we aim to study noncumulative measures that answer the question “In terms of citation impact, what have you done lately?” We single out six measures from the rather sparse literature, including Hirsch’s m-index, a time-scaled version of the h-index. We introduce new measures based on the idea of “citation acceleration.” After presenting several axioms for noncumulative measures, we conclude that one of our new measures has much better theoretical justification. We present a small-scale study of its performance on real data and conclude that it shows substantial promise for future use.

1. INTRODUCTION

Despite strong opinion to the contrary among researchers, it is deemed necessary by bureaucrats worldwide to use simple measures of researcher impact. Measures based on research publications (mostly research monographs and peer-reviewed articles) are heavily used, the most common being the cumulative number of citations N(t), cumulative number of papers P(t), and the h-index h(t) (Hirsch, 2005) (defined as the greatest integer h such that the author has at least h papers each of which has at least h citations). All three quantities above are biased toward senior scholars, being cumulative and therefore automatically increasing over time, even after the end of the researcher’s career. Also, they provide information on overall career citation impact, but no answer to “What have you done lately?” For many purposes it is not particularly useful to know the h-index of Isaac Newton or total number of citations of Albert Einstein. Comparing researchers near the start of their careers, comparing them with more senior researchers, or trying to predict the future productivity and impact of a researcher clearly require different metrics.

Citation metrics that are not automatically increasing have received much less discussion in the literature. For example, the survey (Wildgaard, Schneider, & Larsen, 2014) of 108 author-level metrics contains at most 15 that are not automatically increasing and which attempt to measure time-varying performance. The earlier survey (Bornmann, Mutz, et al., 2011) of variants of the h-index included only six variants that attempted to adjust for career age, out of 37 indicators. Of course, nonincreasing measures intended to account for career age have been covered by some researchers. Indeed, Hirsch in his original paper (Hirsch, 2005) devoted substantial analysis to the rate of growth of h with the number of years t since the author’s first publication, and defined the m-index by m(t) = h(t)/t. He calculated m for a selection of physicists (using a single fixed year, presumably 2005) and concluded that a value of m around 1, 2, 3 correlated with his judgment of distinguishing between “successful scientist,” “outstanding scientist,” and “truly unique” individual respectively. The m-index was immediately studied by others (Burrell, 2007; Liang, 2006) but has been relatively little explored since. Another measure based on the h-index and attempting to measure recent performance is the contemporary h-index (Sidiropoulos, Katsaros, & Manolopoulos, 2007). Other measures based on the h-index and attempting to adjust for career age include the AR-index (Jin, Liang, et al., 2007), the square root of the sum over papers of citations per year, restricted to publications in the h-core (the minimal set needed for computation of the h-index). The literature contains fewer nonincreasing measures not involving the h-index. One could of course also look at the analog of the AR-index where all publications are considered. We stop here, conscious that we must draw a line somewhere—given the axiomatic approach to be taken below, it already seems clear that most of the above measures will fail many of the axioms.

1.1. Our Contribution

We argue that the “instantaneous rate of accumulation of citations owing to recent work” is the relevant measure of recent citation productivity. We claim that this is precisely the “citation acceleration,” the second time-derivative of the number of citations accumulated by an author. As this quantity is not directly observable, we explore measures aimed at approximating it.

These measures are
wt:=2Ntt2
Wt:=Nt2Nt1+Nt2
W5t:=172NtNt12Nt2Nt3+2Nt4.

We single out six existing measures from the literature that are noncumulative and intended to adjust for career age. These are

In Section 3 we evaluate all nine citation measures against axiomatic criteria. The difference between “theoretical” and “empirical” work in bibliometrics has been well described by Waltman and Van Eck (2012). Our approach here is grounded in a theoretical analysis. The axiomatic approach to bibliometric indicators has been applied to the h-index by several authors following the initial paper (Woeginger, 2008)—we single out Quesada (2011) and Bouyssou and Marchant (2013). To our knowledge, axiomatics have been applied to few other indicators. We single out Waltman and van Eck (2009) and Bouyssou and Marchant (2014), which gives axiomatic characterizations of several well-known indicators such as N and P.

To check that we have not strayed too far from reality, we evaluate the performance of the theoretically best measures W and W5 on several small data sets of mathematical researchers. The results, as shown in Section 4, are promising with respect to predictive value and correlation with expert judgment.

1.2. Definition of Remaining Measures

The age-weighted citation rateA(t) is obtained by summing over all publications the average number of citations per year, evaluated at time t. The measure μ(t) is simply the average number of citations per year, evaluated at time t.

The contemporary h-index is defined analogously to the h-index, where citations gradually lose value. The contemporary h-index uses an adjusted weight of citation, and requires the researcher to have published at least h papers, each with adjusted citation weight to be at least h. The adjusted citation weight of a paper published in year t0 and having C citations up to year t is given by
γtt0+1δC,
where δ and γ are fixed positive constants. For example, if γ = δ = 1, the adjusted citation weight of a paper is the number of citations per year since the year before its publication.
The trend h-index is defined using similar reasoning to the contemporary h-index, except that the weight depends on the year of citation, not of publication. Each citing paper in year t contributes a weight of
γtt0+1δ.

2. BACKGROUND AND DEFINITIONS

Consider a researcher emitting research publications starting from time t = 0. These publications accumulate citations at a certain rate, dependent both on such factors as the number of publications, the size of the research field, the citation practices of the field, and the attractiveness of the papers to other researchers. As mentioned above, the most commonly used metrics are

  • • 

    P(t), the number of publications up to time t;

  • • 

    N(t), the number of citations up to time t; and

  • • 

    h(t), the (Hirsch) h-index at time t.

Note that these each increase in t, even after the end of the researcher’s career.

2.1. The Simple Citation Model

For definiteness we measure time in years and t = 0 corresponds to the date of the first publication. We first consider what we call the “simple model” in continuous time. In this case, a researcher publishes at a constant rate of p papers per unit time and each paper attracts citations at a rate of c per paper per unit time, forever. A discrete version of this model was used by Hirsch (2005). Then the number of publications by time t is P(t) = pt, and the total number of citations is
Nt=0tpcsds=pct2/2.
(1)
The “acceleration” in citation accumulation is therefore
Nt=pc.
(2)

Our main idea is that the quantity pc measures the instantaneous accumulation of citations from new work. This “acceleration” is a key measure of recent productivity and citation impact. For a given research field, larger values indicate researchers with greater recent impact.

The instantaneous citation acceleration cannot be measured directly owing to the discreteness of available publication data, so we introduce the quantity
Wδt=Nt2Ntδ+Nt2δδ2,
which is the usual backward difference approximation to the second derivative. When δ = 1 year, this reduces to the measure W defined above.

We also consider the measure W5, obtained by fitting quadratics to N through successive windows of five data points (separated by 1-year intervals) and approximating the second derivative on each. This is an example of a Savitzky-Golay filter, widely used to smooth discrete data of this type. In fact W5 is the simplest such filter, and we could define W2k+1 analogously for k > 2 by using larger numbers of points.

The measure w is clearly constant in the simple model, with value pc. So is Wδ, as shown by
Wδt=pc2t22tδ2+t2δ2=pc.

So is W5, because the filter used is exact for quadratics and hence reproduces the (constant) second derivative.

As explained by Hirsch (2005), in this model we may assume that the papers in the h-core at time t have been published up to time st, so that ps = h(t), while the number of citations of the least cited element of the h-core is c(ts) = h(t). This leads immediately to
ht=pcp+ct,
so that
mt=pcp+c.
As an aside, note that this equals the harmonic mean of p and c.
From above it follows that in the simple model
Nt=p+c22pcht2,
(3)
so that
ht=βNt,
(4)
where β = 2pcp+c. This is the reasoning behind the definition of α1, and clearly in the simple model we have
α1t=2pcp+c.

Thus in the simple model several measures are constant. However, not all constants are equally valid. The units of acceleration should be citation/(year)2, and indeed w, W, W5 have these units. However the units of m and α1 are inconsistent with this requirement.

The value of μ(t) is clearly pct/2. The age-weighted citation measure A(t) equals pct in this model, because in Eq. 1 we divide the integrand by s. The contemporary h-index and trend h-index are harder to compute, but are also nonconstant unless δ = 1 (we use the method shown above for the h-index, and omit the details here).

2.2. A Model Incorporating Retirement

We introduce another simple model in order to analyze the prediction value and behavior of measures when a researcher stops publishing papers due to any reason. In the second model we suppose that publications in the simple model stop after time T. In this case, the number of citations is N(T) = pcT22, while for t > T we have an additional pcT(tT) citations. Thus if tT,
Nt=pcTtT/2.
(5)

Direct computation shows that the measure W takes the value zero provided Tt + 2, so that sufficient time has elapsed for measurements to be taken. However, the other measures above take on nonzero values when t > T. We see that w has the value pcT(2tT)/t2. For example if t = 2T, so the total time elapsed after retirement is as long as the researcher’s entire career, w has reduced by only 25%, to 3pc/4, from its previous constant value pc during the career.

Hirsch’s original argument shows that the h-index has the same value pct/(p + c) in the second model until time t = T(1 + p/c) as it had in the simple model, but that after this time it has value pT/t. Interestingly, this is independent of c. The measure α1 takes the value p3cT when t = 2T. We do not compute the details of the contemporary and trend h-index here, because we can see enough to rule them out axiomatically below.

It is of course possible to explore more complicated models, but the focus of this article is the introduction of a new measure with axiomatic justification, so we now proceed to that.

3. AXIOMS

As mentioned above, the number of possible measures is enormous, but without axiomatic foundations, it seems pointless to study them in detail. We now present several axioms for measures intended to describe noncumulative citation impact. All except the last seem to us to be uncontroversial.

  • • 

    Computability: The measure should be easily computable from citation counts, paper counts, and academic age.

  • • 

    Units: The units of the measure should be (citation)/(time)2.

  • • 

    Locality: If no citations are gained during a time interval, the measure is zero during that interval.

  • • 

    Constancy: In the first model, the measure is constant.

  • • 

    End of career: In the second model, the measure is zero for t > T.

  • • 

    Packaging-independence: The measure should not depend on P: It is computable only from citation counts and academic age.

The Packaging-independence axiom requires more explanation. We argue that the impact of a researcher with 10 papers each attracting 100 citations is the same as if all 10 papers had been combined into a book that receives 1,000 citations. The packaging into publications may in practice affect the number of citations in a more complicated way, but if publications have no overlap and citations reflect intellectual influence only, this should not occur. Of course, this is also a strong argument against using the h-index.

The Locality axiom may need to be interpreted slightly differently when time is discrete. For example, the measure W satisfies this axiom provided the point chosen is 2 years past the left endpoint of the interval in question.

Table 1 shows the performance of the abovementioned measures against these axiomatic criteria. Clearly, W and W5 perform much better than the others, and we consider only these measures in the next section. Note that hc requires specification of two free parameters before we can even compute it, and we know of no principled way to do that, hence the failure of the Computability axiom. The other axioms were evaluated for arbitrary γ and δ, and give the same answer for all choices of these constants (with an exception for the constancy axiom, as noted above).

Table 1. 
Performance of measures with respect to axioms
AxiomMeasure
wWW5mα1hchtAμ
Computability ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✓ ✓ 
Units ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ 
Locality ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ 
Constancy ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ 
End of career ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ 
Packaging-independence ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ 
AxiomMeasure
wWW5mα1hchtAμ
Computability ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✓ ✓ 
Units ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ 
Locality ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ 
Constancy ✓ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ 
End of career ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ 
Packaging-independence ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✓ 

Based on the discussion above, we move on to consider only W(t) and W5(t), which seem the most promising measures.

4. CASE STUDY OF MATHEMATICAL RESEARCHERS

We now discuss the performance of the measures on real data.

4.1. Methods

The experiments described below were carried out in February 2020. All raw data and computed data are publicly available (Wilson, 2020).

We concentrated for this article on the research area most familiar to us, namely mathematics. We compiled several data sets based on specific sets of researchers. The lack of availability of this data in an open format, or even in a proprietary one that allows for comprehensive analysis, is a major factor in hampering studies of this type. We resorted to making many time-intensive manual web-based queries. To test our hypotheses, for each researcher we require the total number of citations to his or her works in each calendar year. We used Web of Science Core Collection, which we chose because of its availability and reasonably wide coverage.

We extracted four sets of mathematicians and categorized them. The first set used consisted of all Abel Prize winners still living at May 1, 2018 (with the exception of two for whom author name disambiguation was too difficult) and included 17 authors. The second consisted of 10 mathematicians from a single department (University of Massachusetts Amherst, Mathematics & Statistics; UMass) with interests in a common subfield (algebraic geometry). The third data set consisted of 10 authors generated “randomly” from MathSciNet (we chose authors of the most recent papers in algebraic geometry according to MathSciNet). The fourth consisted of all living winners of the Fields Medal from 2006 to 2018 inclusive, and consisted of 13 authors. This gave 50 mathematical researchers. For each researcher we took year 1 to be the first year t for which N(t) ≥ 10.

Larger data sets would give more confidence in the results below, but they are clear enough to show that the axiomatically well-founded measures W and W5 measure something about a researcher that enables us to distinguish between randomly chosen, successful and outstanding researchers.

4.2. Results

4.2.1. Variation of Measures

In Figure 1 we graph, for three Abel and three Fields prizewinners, W(t) and W5(t) for t from 5 years after career start (defined as the first year for which N exceeds 10). As can be seen, there is substantial variation over time for each author. In Table 2 we give the mean and standard variation of the values of the measures over the same time period.

Figure 1. 

Variation over time of values of W and W5 for three selected Abel and Fields prizewinners.

Figure 1. 

Variation over time of values of W and W5 for three selected Abel and Fields prizewinners.

Table 2. 
Mean and standard deviation for W and W5 for Abel prizewinners
 mean(W)sd(W)mean(W5)sd(W5)
Serre, JP 1.00 4.05 −0.36 2.04 
Atiyah, MF 9.27 34.88 10.73 14.95 
Singer, IM 1.91 4.21 0.56 1.20 
Lax, PD 5.91 19.11 5.14 2.94 
Carleson, L. 1.45 8.49 1.14 2.67 
Varadhan, SRS 2.45 27.89 6.10 5.85 
Tits, J 1.09 7.81 0.90 2.01 
Tate, JT 1.45 6.36 −0.43 2.27 
Milnor, J 3.91 11.08 3.64 5.99 
Szemeredi, E 2.45 7.94 2.19 1.24 
Deligne, P 0.09 2.39 −0.08 0.87 
Sinai, YG 0.55 2.46 0.17 0.60 
Nirenberg, L 3.36 8.71 2.95 1.86 
Wiles, A. 2.73 8.43 1.09 2.23 
Meyer, Y 9.64 12.17 5.94 5.63 
Langlands, RP 0.82 5.42 0.99 1.21 
Uhlenbeck, K 2.82 18.70 5.78 5.45 
 mean(W)sd(W)mean(W5)sd(W5)
Serre, JP 1.00 4.05 −0.36 2.04 
Atiyah, MF 9.27 34.88 10.73 14.95 
Singer, IM 1.91 4.21 0.56 1.20 
Lax, PD 5.91 19.11 5.14 2.94 
Carleson, L. 1.45 8.49 1.14 2.67 
Varadhan, SRS 2.45 27.89 6.10 5.85 
Tits, J 1.09 7.81 0.90 2.01 
Tate, JT 1.45 6.36 −0.43 2.27 
Milnor, J 3.91 11.08 3.64 5.99 
Szemeredi, E 2.45 7.94 2.19 1.24 
Deligne, P 0.09 2.39 −0.08 0.87 
Sinai, YG 0.55 2.46 0.17 0.60 
Nirenberg, L 3.36 8.71 2.95 1.86 
Wiles, A. 2.73 8.43 1.09 2.23 
Meyer, Y 9.64 12.17 5.94 5.63 
Langlands, RP 0.82 5.42 0.99 1.21 
Uhlenbeck, K 2.82 18.70 5.78 5.45 

4.2.2. Predictive Value of Measures

As explained by Penner, Pan, et al. (2013), cumulative increasing measures such as the h-index contain intrinsic autocorrelation, which vastly overstates their predictive power. They find that the actual ability of the h-index to predict future citations from future publications is rather low. In our case, we are dealing with noncumulative measures, whose predictive power is not so clear. The results of Section 4.2.1 show considerable variation in the year-to-year values of W(t) and W5(t), so the idea of the simple model, that these measures are constant and hence precisely determine something intrinsic to the researcher, is not plausible. We do not expect to be able to predict the value of W(t + 5), for example, from W(t) only. However, the results in Section 4.2.3 show that gross distinctions between researchers at different levels of impact can be made (when dealing with researchers in the same fairly narrowly defined field), and these seem to mean something.

To obtain a better idea of predictive power, for the union of our data sets we computed the mean in years 3–5 of career of W, and used this to attempt to predict the mean in years 6–8 of the same measure. The ordinary least squares linear regression results are displayed in Figure 2 (note that the extreme outlier Terence Tao was removed from the data set, as was the very young Peter Scholze, leaving 48 researchers). The value R2 = 0.748 shows a high level of predictive power. Note that the definition of W means that we are trying to predict a linear combination of N(8), N(7), N(5), and N(4) from a linear combination (with the same coefficients) of N(5), N(4), N(2), and N(1), and there is no reason a priori to expect this to have such a high coefficient of determination.

Figure 2. 

Regression of mean of W in years 4–6 against mean in years 1–3.

Figure 2. 

Regression of mean of W in years 4–6 against mean in years 1–3.

4.2.3. Relation to Expert Judgment

We expect that the citation acceleration for randomly chosen authors should be lower overall than that of the UMass researchers. Also, we expect the Fields data set and Abel data set to have higher values of the measures than the UMass data set. Given the age of the members of the Abel data set, we expect a small advantage to the Fields data set. All our expectations are borne out by the results shown in Figure 3.

Figure 3. 

Scatterplot of values of W for four different data sets.

Figure 3. 

Scatterplot of values of W for four different data sets.

5. DISCUSSION

5.1. Limitations

Before summarizing our main points, we acknowledge that the entire subject of bibliometrics is suspect in the eyes of many researchers, because of its perceived misapplication by unimaginative bureaucrats having influence over researcher careers. We share these concerns, and the present article was motivated by the idea that if we researchers are forced to be evaluated by simple metrics, we can at least have some agency in their design. The popularity of the h-index, for example, is very mysterious to us, given its weak theoretical foundations (Waltman & Van Eck, 2012). Any one measure can be strategically manipulated, but we hope that by use of sufficiently many metrics with low theoretical correlation, incentives for researchers to act in ways that are not helpful to science overall will be reduced.

All of the citation measures in the literature are susceptible to many problems (including missing data, author name disambiguation, negative citations, contributions of multiple authors, citation inflation owing to growth in number of researchers). Also, there is the problem of normalization across different fields (for example, one citation in mathematics corresponds roughly to 19 in physics and 78 in biomedicine [Podlubny, 2005]). This question of “field” is a difficult one—it is obvious that certain areas of mathematics have communities of different sizes, leading to substantial variation in the number of citations across areas. Thus, as usual, there is no completely automated substitute for human judgment.

5.2. Positive Outcomes

The index W(t) introduced in this article seems to measure something specific to a researcher that is related to his or her recent productivity and impact, and seems promising as a way to make coarse distinctions between researchers in the same field who may be at different career stages. It behaves well with respect to natural axioms. It seems fairly well correlated with subjective measures of research impact or quality. It is less sensitive to the way in which ideas are packaged into individual publications, and considerably easier to compute, than the m-index (under the assumption that splitting a paper splits the citations in the obvious way, the m index discourages extreme “salami-slicing,” whereas W is indifferent to it and P encourages it).

Insofar as citation metrics are to be increasingly used for evaluation of researchers and especially for allocation of resources to them, the W-index provides another useful (perhaps the single most useful found so far) measure of recent publication activity leading to citation impact, and one that has decent predictive value.

5.3. Future Work

We are grateful to an anonymous referee for informing us about two papers that deal with the issue of discrete data in bibliometrics (Liu & Rousseau, 2012, 2014). The idea used in those papers, to approximate (in their case the citations to a single research article) by a continuous cubic spline, could also be used in the situation of the present paper. For example, we could approximate N(t) by a cubic spline and then estimate the second derivative by taking the second derivative of the spline. The measure so derived would satisfy all our axioms. We do not expect substantially different results from using interpolation rather than filtering as we have done, but it may be worth further investigation.

Bouyssou and Marchant (2014) state that their paper explicitly does not deal with any indicators intended to adjust for career age, and the last part of the paper suggests further work in such a direction. We offer the present work as an initial contribution, and intend to follow up. A stream of research initiated by Woeginger (2008) deals with axiomatic characterizations of the h-index—that is, a set of axioms that, taken together, uniquely determine the h-index. Our experience with characterization theorems (and also impossibility theorems where “too many” axioms are chosen, a prominent approach in social choice theory, for example) is that very often the axiom systems consist of a few innocuous assumptions and one that is much less intuitive and essentially encodes the desired result. Nevertheless, it would be interesting to obtain an axiomatic characterization of our measure W, for example.

The relation of Eq. 4 derived from the simple model has wider validity than might be expected at first sight. Mannella and Rossi (2013) find via a study of 1,400 Italian physicists that this quadratic relationship holds well on real data, and empirically find the best fit value of β = 0.53 in Eq. 4, agreeing with the rough calculations of Hirsch based on a smaller data set of physicists. Yong (2014) showed analytically, based on theory of random partitions, that a very good estimate should be β = 6 ln 2/π ≈ 0.54. He also demonstrated the accuracy of this approximation on a small data set of prominent mathematicians.

However, Mannella and Rossi also showed that the time-scaled index
α2t=htt
(6)
is approximately time independent on their data set. This implies that the number of citations is better described as linear rather than quadratic, which is clearly inconsistent with the simple model. Attempts to estimate citation acceleration depend on the quadratic growth of citations with time, so a possible linear relationship will have an effect on the measures used in this paper: In that case, the acceleration would be identically zero. We feel that the growth of citations by an author with time should be studied seriously on much larger data sets than we have treated here.

To concentrate on the main concept of citation acceleration, we have omitted more subtle issues, such as weights for coauthored papers and normalization by the size of the research field. These of course could be explored.

AUTHOR CONTRIBUTIONS

Mark Wilson: Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Supervision, Writing—original draft, Writing—review & editing. Zhou Tang: Formal analysis, Software, Visualization.

COMPETING INTERESTS

The authors have no competing interests.

FUNDING INFORMATION

This project received no funding.

DATA AVAILABILITY

Data are available from Harvard Dataverse.

ACKNOWLEDGMENTS

We thank Hooman Alavizadeh for preliminary discussions and Thierry Marchant for feedback on a draft of this article. We thank the editor and referees of this journal for their constructive comments on the initial submission.

REFERENCES

Bornmann
,
L.
,
Mutz
,
R.
,
Hug
,
S. E.
, &
Daniel
,
H.-D.
(
2011
).
A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants
.
Journal of Informetrics
,
5
(
3
),
346
359
.
Bouyssou
,
D.
, &
Marchant
,
T.
(
2013
).
An interpretable axiomatization of the Hirsch-index
. In
14th ISSI Conference
(pp.
2024
2026
).
Leuven
:
International Society for Scientometrics and Informetrics
.
Bouyssou
,
D.
, &
Marchant
,
T.
(
2014
).
An axiomatic approach to bibliometric rankings and indices
.
Journal of Informetrics
,
8
(
3
),
449
477
.
Burrell
,
Q.
(
2007
).
Hirsch index or Hirsch rate? Some thoughts arising from Liang’s data
.
Scientometrics
,
73
(
1
),
19
28
.
Hirsch
,
J. E.
(
2005
).
An index to quantify an individual’s scientific research output
.
Proceedings of the National Academy of Sciences
,
102
(
46
),
16569
16572
.
Jin
,
B.
,
Liang
,
L.
,
Rousseau
,
R.
, &
Egghe
,
L.
(
2007
).
The R-and AR-indices: Complementing the h-index
.
Chinese Science Bulletin
,
52
(
6
),
855
863
.
Liang
,
L.
(
2006
).
h-index sequence and h-index matrix: Constructions and applications
.
Scientometrics
,
69
(
1
),
153
159
.
Liu
,
Y.
, &
Rousseau
,
R.
(
2012
).
Towards a representation of diffusion and interaction of scientific ideas: The case of fiber optics communication
.
Information Processing & Management
,
48
(
4
),
791
801
.
Liu
,
Y.
, &
Rousseau
,
R.
(
2014
).
Citation analysis and the development of science: A case study using articles by some Nobel prize winners
.
Journal of the Association for Information Science and Technology
,
65
(
2
),
281
289
.
Mannella
,
R.
, &
Rossi
,
P.
(
2013
).
On the time dependence of the h-index
.
Journal of Informetrics
,
7
(
1
),
176
182
.
Penner
,
O.
,
Pan
,
R. K.
,
Petersen
,
A. M.
,
Kaski
,
K.
, &
Fortunato
,
S.
(
2013
).
On the predictability of future impact in science
.
Scientific Reports
,
3
(
1
),
1
8
.
Podlubny
,
I.
(
2005
).
Comparison of scientific impact expressed by the number of citations in different fields of science
.
Scientometrics
,
64
(
1
),
95
99
.
Quesada
,
A.
(
2011
).
Further characterizations of the Hirsch index
.
Scientometrics
,
87
(
1
),
107
114
.
Sidiropoulos
,
A.
,
Katsaros
,
D.
, &
Manolopoulos
,
Y.
(
2007
).
Generalized Hirsch h-index for disclosing latent facts in citation networks
.
Scientometrics
,
72
(
2
),
253
280
.
Waltman
,
L.
, &
van Eck
,
N. J.
(
2009
).
A taxonomy of bibliometric performance indicators based on the property of consistency
(
Tech. Rep. No. ERS-2009-014-LIS
).
Rotterdam
:
Erasmus University
.
Waltman
,
L.
, &
Van Eck
,
N. J.
(
2012
).
The inconsistency of the h-index
.
Journal of the American Society for Information Science and Technology
,
63
(
2
),
406
415
.
Wildgaard
,
L.
,
Schneider
,
J. W.
, &
Larsen
,
B.
(
2014
).
A review of the characteristics of 108 author-level bibliometric indicators
.
Scientometrics
,
101
(
1
),
125
158
.
Wilson
,
M. C.
(
2020
).
Replication Data for: Non-cumulative measures of researcher citation impact
.
Harvard Dataverse
. https://doi.org/10.7910/DVN/SQE29Z
Woeginger
,
G. J.
(
2008
).
An axiomatic characterization of the Hirsch-index
.
Mathematical Social Sciences
,
56
(
2
),
224
232
.
Yong
,
A.
(
2014
).
Critique of Hirsch’s citation index: A combinatorial Fermi problem
.
Notices of the AMS
,
61
(
9
),
1040
1050
.

Author notes

Handling Editor: Ludo Waltman

This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. For a full description of the license, please visit https://creativecommons.org/licenses/by/4.0/legalcode.