The gendered structure of science does not transpire in an experimental vacuum

https://www.webofscience.com/api/gateway/wos/peer-review/10.1162/qss_c_00300

An article recently published in Quantitative Science Studies proposed an apparently elegant solution to go past the alleged limits of “observational approaches” and “simple statistical associations” when it comes to the study of gender disparities in scientific funding (Cruz-Castro & Sanz-Menéndez, 2023, p. 594). A randomized experiment, it was claimed, would finally allow us to “measure the effects of a cause,” that is “the effect of the gender of the principal investigator (PI) on the score of a research funding application” (Cruz-Castro & Sanz-Menéndez, 2023, p. 594). And the finding of this trial was as clear as its formulation: There was no indication whatsoever that women’s applications had been evaluated differently than those of men.

Although we can certainly be delighted that evidence of gender discrimination could not be found, it is doubtful that experimental approaches will bring any clear understanding of the sociological mechanisms that make up the gendered structure of science. In this letter, we underline three main issues that render this method ill suited for studying the scientific field’s functioning. Although we use the experiment performed by Cruz-Castro and Sanz-Menéndez as the base for our discussion, we do not think that these problems are specific to this work and invite readers to consider more critically what we can gain—or, most likely, lose—scientifically from experimental methods.

The first limit that experimental approaches face is that they resort to scenarios that are at best a crude replica of the actual scientific processes they intend to shed light on. Hence, to gauge gender bias in funding, Cruz-Castro and Sanz-Menéndez asked reviewers working for a Spanish funding agency to “score a hypothetical application based on the description of some attributes relevant for the assessment, with the same definition of the evaluation criteria, using an evaluation template identical to the one used by the funding agency” (Cruz-Castro & Sanz-Menéndez, 2023, p. 606). Not only did the participants know that the application was not real—which obviously reduces the stake that is at play when we distribute funding—but the applications themselves were rendered symbolically meaningless because they contained no information whatsoever about applicants’ positions within the scientific field (Bourdieu, 2004). Applicants had a gender, but no discipline, no institution, no research topics, no theoretical orientations. Even their CV is inscrutable: Instead of referencing articles in scientific journals that signaled something to reviewers (in terms of scientific positioning, prestige, orthodoxy, etc.), Cruz-Castro and Sanz-Menéndez decided that participants would only have knowledge about the “number,” “type,” and “impact indicators” of their publications (Cruz-Castro & Sanz-Menéndez, 2023, p. 606), as if such disembodied elements sufficed.

Another issue in experimental studies of the scientific field lies in the linear model that underpins many of the hypotheses that are derived from them. Aiming for statistical causality and trying to estimate the “true” net effect of the variables they manipulate, Cruz-Castro and Sanz-Menéndez thus purposefully disconnect gender from its social context of expression. The gender of applicants and reviewers comes to stand on its own, as if it was not part of, and giving order to, a broader social structure in which it is intertwined with a number of social mechanisms that legitimate, reproduce, and extend but also, sometimes, reverse, its effects (Connell, 2005; Harding, 1986). Again, this does not reproduce the reality of the scientific field. Whether in funding, hiring, or publication decisions, outcomes can become gendered—and, more often than not, end up favoring masculine positionings—because they are always interacting and entangled with social judgments of excellence (Orupabo & Mangset, 2022), symbolic hierarchies (Larregue & Nielsen, 2024), and devotion to work (Rivera, 2017). To put it otherwise, gender disparities in science are, possibly to a large extent, reproduced in part because they are inscribed in judgments that are seemingly neutral and partially aligned on the rules of the game of the scientific field. It is thus not surprising that an experiment isolating gender from this broader context would find no sign of disparities: The very mechanisms leading to such inequalities have been neutralized.

As Cruz-Castro and Sanz-Menéndez underlined, research on gender disparities in scientific funding has led to somewhat contradictory findings. Although they acknowledge that this might be due in part to the different countries, disciplines, and contexts in which this issue was investigated, the core message of their article is that this lack of consistency can be explained by the “methodological and measurement shortcomings” of nonexperimental approaches (Cruz-Castro & Sanz-Menéndez, 2023, p. 615). Although it is quite likely that some studies on gender inequalities in science do present methodological shortcomings, we have seen that experimental designs are definitely not exempt from critique. More importantly, by trying to solve the apparent contradictions in the literature through supposedly more robust causal measurements, Cruz-Castro and Sanz-Menéndez obscure the need for better explanations, including through comparative work between countries, disciplines, and funding agencies. It is not surprising at all that gender and scientific funding do not interact in the same way everywhere, as these are not universal but social categories that are shaped and reshaped across time and space.

Julien Larregue: Conceptualization, Funding acquisition, Supervision, Validation, Writing—original draft, Writing—review & editing. Hassina Bourihane: Writing—original draft.

The authors have no competing interests.

This research is funded by the SSHRC (grant number 435-2023-0882) and FRQSC (grant number 328913).