Towards a moralization of bibliometrics? A response to Kyle Siler

In a recent letter to QSS, Kyle Siler (2021), made harsh comments against the decision of the editors to publish a controversial paper signed by Alessandro Strumia (2021) about gender differences in high-energy physics. Though, like him, “I would rather be doing other things than writing this letter,” I think it is important not to pass over this letter in silence as it raises important questions about what I see as a recent and very problematic trend in scholarship: the tendency to replace rational and technical arguments with a series of moral statements and ex cathedra affirmations that are not supported by cogent arguments.

My aim here is not to add another layer of technical comments on Strumia’s paper but to point to the elements in Siler’s letter that are typical of this new tendency that now also begins to affect debates within the bibliometric research community.

The most striking feature of Siler’s letter is the moral high horse from which he formulates his many questionable statements, which are often based on insinuations about supposed “questionable values” or his being “disappointed” by decisions he disapproves. According to him, Waltman allegedly offers “vacillating”’ justifications for publishing a “bad” paper, and even shows “stubbornness” and “arrogance.”

Though it is reassuring that Siler adds that “there are many good people affiliated with QSS,” we are not told what the basis is for his strong convictions other than that he just happens to dislike the paper. But anyone with a modicum of understanding of the dynamics of scientific research would not be surprised that papers are often “widely criticized.” And invoking the Sokal Affair shows a surprising misunderstanding on his part of the nature of the debate surrounding Strumia’s paper, which is not a hoax! The nearly hysterical tone of the letter is also apparent in his comparison of Strumia’s paper with “Flat Earthers” and his bizarre references to African Colonialism, eugenics, and LGBT parents! Another example of the ignorance of the nature of publishing is Siler’s character assassination and talk of “cowardice” for not publishing the review reports. It should be obvious that editors cannot do that without the permission of the authors. And why should experts in bibliometrics be impressed by the fact that “thousands of physicists signed a petition condemning Strumia’s work and ideas”? Since when is a petition an argument? I will not comment on the fact that the very speed at which the petition was signed probably means that most of the signatories never analyzed the data and were simply reacting emotionally.

The moral tone of Siler’s indictments is also evident in the fact that he seems to consider that “the implications of the article” which could be “damaging” (for whom?) are more important than its empirical validity. After all, suppose some bibliometric results published by two female researchers using the usual valid methods would somehow contradict the personal moral convictions of some activist scholars about, say, “equity”? Should the results be burned? Retracted if published? Seemingly having the capacity of a medium, Siler even affirms that having published Strumia’s paper is “a bad omen for QSS on gender and science issues”! And as Siler mentions the case of the retracted paper by Nature Communications in November 2020, let us look into that case more carefully, as it is very instructive. That controversy also reflects a moralizing trend that is detrimental to rational discussions of empirical results.

In a paper titled “The association between early career informal mentorship in academic collaborations and junior author performance,” the authors (two women and one man), found that “mentorship quality predicts the scientific impact of the papers written by protégés post mentorship without their mentors” (AlShebli, Makovi, & Talal Rahwan, 2020a). They also found that “increasing the proportion of female mentors is associated not only with a reduction in post-mentorship impact of female protégés, but also a reduction in the gain of female mentors.” In the weeks following the publication, and without having the possibility to closely analyze the complex method used by the authors, these conclusions have been read by many as a “critique” of female mentorship, and social media hysteria led to the retractation of the paper. Now, let us consider the reasons invoked by the authors to justify retracting their paper. While reading them, keep in mind that the reasons to retract a paper are usually related to the fact that the results are somehow flawed (fake or manipulated data, lack of reproducibility, plagiarism, etc. [Shi, Wang et al., 2020]). So, here is the explanation provided by the authors (AlShebli, Makovi, & Talal Rahwan, 2020b):

What is striking here is that they talk about criticism of their assumptions and interpretations. But criticism is usually the ground for discussion, not retraction, right? They then add

So, “although” they consider “all the key findings” valid, they still conclude that the “most appropriate” action is to retract the paper! Here again, that kind of debate is usually found in letters and rejoinders criticizing the interpretation or method, and do not lead to retraction. But I think we better understand what is really going on here when we read the last paragraph of their explanation:

One can of course appreciate that the authors express their “steadfast solidarity” with women and that they have strong feelings about “equity,” but who could seriously be against equity? These are certainly laudable moral statements expressing publicly their personal convictions, but how are they related to the method and technical content of the paper? And how do these moral convictions justify retracting a paper when the authors clearly believe that their findings are valid? And how is that compatible with their hope that academic debate continues and should “thrive on robust and vivid scientific exchange,” when their action is the very negation of such debates?

It is obvious to me that the authors have in fact caved in to moral pressures instead of better explaining that their assumptions and results were warranted after all. For just suppose, for the sake of the argument, that the scientific field is male dominated and that female scientists are indeed kept in subjugated positions. In such a world, it cannot be a surprise to find that a woman mentored by a man will have a greater chance of being integrated in a central network than a woman mentored by a woman who is part of the subjugated group and thus has a less powerful network. Finding the contrary would in fact be surprising. Of course, the details of their methodology are complex and can always be discussed, but their result does not come as a surprise. In light of this brief analysis, here is my explanation of that controversy: Those who, for moral and ideological reasons, do not like to read such a conclusion, will try to demolish the methods. Now, suppose that the same method had found the opposite result: Being mentored by women gives rise to a larger “scientific impact” than being mentored by men. Would that have led to a scandal and social media hysteria? I bet not. And here is an empirical argument supporting my prediction: The very same authors, using essentially the same complex method, published a paper in the same journal in 2018 about “The preeminence of ethnic diversity in scientific collaboration” and concluded that “ethnic diversity had the strongest correlation with scientific impact” (AlShebli, Rahwan, & Woon, 2018). This is a nice conclusion as we are certainly happy to learn that “ethnic diversity” is not only a good thing in itself but that it also leads to better science! So, nobody ever complained about that paper despite its complex method of identifying “ethnicity, discipline, gender, affiliation, and academic age.”

What all this calls to mind is the classic “confirmation bias,” which implies that we are much less critical of results we like than of results we don’t like. The difference now is that bias is expressed through moral injunctions and bullying by self-appointed moral entrepreneurs instead of arguments.

The content and tone of Siler’s letter is part and parcel of this new trend and one can seriously doubt that it can lead to producing better and more valid results that really shed light on existing inequalities.

So, far from criticizing the editors for having created a space for rationally debating Strumia’s paper, we should congratulate them for having done so. QSS should thus stand for robust academic criteria and reject moralizing comments containing personal attacks or insinuations about the supposed hidden motives of the paper criticized. Editors should simply ask the critics to stick to arguments directly related to the content of papers discussed. This is the price to pay to preserve the autonomy of scientific research, keeping it free from any extrinsic endeavors to moralize, be it from churches, states or pressure groups convinced that they embody the only valid version of “social justice.”

For, as the old proverb says: “The road to hell is paved with good intentions.”