Cycling on the Freeway: The perilous state of open-source neuroscience software Open Access

The development and maintenance of open-source scientific software is labor-intensive, especially when adopting best practices of open science: readability, resilience, and re-use (Connolly et al., 2023). Here (and throughout) when we say “Open-source software” or “FOSSS,” we are referring to larger software packages, not the openly shared project code of single researchers or research teams. While the latter is an important and useful contribution to Open Science, the former comes with unique challenges regarding project administration and management, software maintenance (often across different operating systems), and catering to a large user base. Critically, the survival of any open-source software is guaranteed not only by the total number of contributors, but also by the number of developers who have a sufficiently complete overview of the source code and the tools and processes in place for maintenance of the project. The Truck Factor (or Bus Factor) quantifies the number of such maintainers for a given project, and can be seen as an indicator of the risk of incomplete knowledge- and capability-sharing among the project’s team members (Avelino et al., 2016). Box 1 shows the Truck Factors of several widely used analysis software projects in electrophysiological neuroscience. It is evident that projects typically have a Truck Factor of one to three, meaning that only one to three people per project have sufficient knowledge to keep the project alive (note that the Truck Factor does not take into account whether those contributors with the relevant knowledge are still active or have funding/support to do the maintenance work). This reveals the fragility of the academic open-source system: all users of a software package rely on the work of one to three people to keep everyone’s data analyses from failing. It also highlights the enormous pressure the maintainers of open-source software are under. Given the importance of scientific software to the practice of modern science, one would hope developing FOSSS would be a well-supported and incentivized role in academia. This hope is not borne out: the academic incentive structure for software development and maintenance is woefully inadequate (Carver et al., 2022; Davenport et al., 2020; Merow et al., 2023; Smith et al., 2018) and can even be perceived as hostile (Millman & Pérez, 2018; Pérez, 2011). As Davenport et al. (2020) state: “In spite of the vital role research software plays, it largely remains undervalued, with time spent in training or development seen as detracting from the ‘real research’.” As maintainers and developers of neuroscience FOSSS (see caption of Box 1 for details) and researchers of the open-source movement in academia, we strongly agree with this notion: The current incentive model often summarized as “publish-or-perish” has a significant negative impact on the software ecosystem’s health in several ways, which we will describe below.

Many scientists who develop software publish a paper to introduce their project to the community, and direct users to cite that paper if they use the software for their analysis (Katz & Chue Hong, 2018; Smith et al., 2016). However, software papers are usually published to mark the first public release or sometimes a later major release of a software package. Given that publication counts are such an important (and often the primary) metric for academic tenure and promotion, once the paper announcing the software is published, there is little incentive from the employer to continue developing the software. This may be changing as academic institutions rely more on composite citation metrics rather than simple publication counts—by incentivizing that the paper be widely cited—but the reward of new projects and attendant new publications can easily outweigh the reward of garnering more software-paper citations, especially given that software papers are viewed as less valuable than research papers (Davenport et al., 2020). At the same time, people who contribute to development or maintenance of FOSSS at a later stage do not get rewarded by citations of the initial paper (Davenport et al., 2020), thus increasing the chance that the software will become unmaintained. We worry that the lack of benefit for project “latecomers” also incentivizes starting new projects instead of contributing to existing ones—since a new project means a new publication—which, in turn, risks further increasing the number of unmaintained software packages in the field. To spell it out: open-source work done by academics (in the time they might otherwise use to do research) sustains other academics writing their papers (Merow et al., 2023)—which puts the software-developing academics behind in the publish-or-perish culture of academia.

Funding is hard to obtain for the development of new software—and even harder for the maintenance of existing software (Davenport et al., 2020). For example, currently in the US National Institutes of Health (NIH) funding system, proposals to continue development of open software are reviewed alongside new-software proposals and are ranked on “Innovation”, which leads to uncompetitive scores for maintenance projects. That means that many contributors and maintainers either do not have secure long-term academic positions or are not paid primarily for their software work. This is slowly changing: more and more programs¹ are supporting the development of FOSSS. However, such grants are often promoting novelty, that is, the addition of new features or the launch of new software projects, not the maintenance of existing projects. Once the grant is over, the primary incentive to continue maintaining the software is largely gone, a void quickly filled by the ever-present publish-or-perish incentive lurking in the background.

Research shows that diversity has positive effects on project outcomes generally (Earley & Mosakowski, 2000; Hoogendoorn et al., 2013; Jackson & Joshi, 2004; Roberson, 2019) and for open-source projects in particular (Daniel et al., 2013; Vasilescu, Posnett, et al., 2015). However, structural factors such as misogyny and racism hinder diversity of the developer pool: only 1.1–5.4% of open-source developers are perceptible as or identify as women (Eghbal, 2016; Geiger, 2022; Ghosh et al., 2002; Nafus, 2012) and less than 17% are perceptible as Non-White (Nadri et al., 2021). The cost of contributing is higher for minorities (Whitaker & Guest, 2020): not only do they often feel a pressure to have to be perfect (Singh & Bongiovanni, 2021), but also members of underrepresented groups have to face stereotyping, discrimination, and harassment in the open-source software world (Frluckaj et al., 2022; Nadri et al., 2021; Nafus, 2012; Singh & Bongiovanni, 2021; Vasilescu, Filkov, & Serebrenik, 2015).

We are unaware of any systematic studies examining diversity in neuroscience software, but our collective experience in the field suggests that it is not substantially different from the broader open-source software community. If true, this is unsurprising: academics need a certain level of privilege to be able to contribute to open source in the first place. Typical barriers to participation in FOSSS projects are the lack of permission and support from a supervisor, the lack of “free” time outside normal work hours due to family or financial demands on that time, and lack of confidence due to inadequate training, role models, and guidance. These barriers all affect marginalized groups more strongly, and are compounded by the lack of representation and role models from underrepresented demographics and by the myth of meritocracy (Nafus, 2012). Moreover, the “informalization” of open source (Nafus, 2012) compounds the problem by eschewing traditional application and advancement processes and legal workplace protections in the spaces and interactions where FOSSS work is carried out, and consequently such spaces are often dominated by “old boys” networks where again underrepresented identities face an uphill battle. Diversity in age and career stage of FOSSS contributors and maintainers is also an issue. Many academic open-source contributors are pursuing or have just finished their Ph.D. and the devaluing of software work makes it harder for them to achieve tenured positions, while more senior academics who contributed early in their career are often not able to prioritize open-source work anymore, due to the publish-or-perish culture and lack of stable funding for software development in academia.

Each of these barriers to achieving a diverse community of FOSSS contributors and maintainers exacerbates the substantial lack of qualified labor (cf. Box 1).

In summary, workplace incentives in academia disfavor participation in open-source communities; the lack of a level playing field further discourages participation for certain classes of academics; and changing the incentives through extramural funding has so far been a temporary, partial fix on too small a scale to be transformative. But given the shift toward relying on FOSSS in neuroscience, it seems clear that the incentive structure must continue to evolve. In the following, we will discuss ways in which such change can be realized.

One obvious cornerstone for a more sustainable and professional FOSSS ecosystem is funding, especially funding that supports maintenance of existing, widely-used projects (Merow et al., 2023). That funding should support scientists who engage in software development as part of their normal academic appointment duties. It has further been suggested that all levels of software development in academia (from widely-used packages down to single-user analysis scripts) can be carried out or supported by experienced research software engineers (Chapuis & Winter, 2024; Connolly et al., 2023; Merow et al., 2023). As others have noted, there are at least two obstacles to academic software work being carried out by professional software engineers: first, academia struggles to attract technically skilled professionals given the (much) higher compensation available in industry (Connolly et al., 2023; Gewin, 2022; Seidl et al., 2016), and second, there is a perception among professional engineers that career advancement prospects in academia are limited (Carver et al., 2022; Connolly et al., 2023; Merow et al., 2023). However, even if institutional support for research software engineering were to increase dramatically, we see a further obstacle to the sustainability of FOSSS: academic software development needs the participation of scientists that are active in the field and have the domain knowledge necessary to understand relevant use cases, best practices, and common analysis pitfalls. This latter point also underscores the importance of having more senior academics involved in software development, which (as discussed above) is disincentivized and, consequently, rare. Thus, we believe that a robust solution must involve institutions increasing their support of working scientists who develop software (perhaps alongside professional research software engineers), and recognizing and rewarding software contributions as an integral part of the practice of modern science.

Increasing the valuation of open-source work for academic career advancement is crucial. One previously suggested, yet debatable, shortcut to this would be the exploitation of the existing publish-or-perish culture by introducing “update publications” after software development milestones (Merow et al., 2023). However, in our view this risks perpetually re-creating the same set of problems, just on a shorter time-scale. Encouraging researchers to cite the software they use—and to cite the software itself (i.e., the specific version used), not the canonical paper that introduced the software—seems like a better adaptation within the existing culture: it does not require the developers to write a new paper, and all contributors to a specific version of the software get credit. Indeed, “cite the software, not the paper” is considered best practice already (Katz et al., 2020; Smith et al., 2016), and we hope that this can become standard practice in neuroscience; developers can make this easy by assigning persistent identifiers like DOIs to each release using services like Zenodo, but publishers, journals, editors, and reviewers must play a role here in insisting that authors attribute their software usage in line with those best practices.

We urge funding bodies and promotion committees to value substantial open-source software work in their evaluation guidelines as an important contribution to science of equal value as one or (ideally) multiple papers. The development of widely used analysis software should be acknowledged and rewarded as a contribution to science rather than viewed as merely the development of one’s coding skills. Individual investigators can play an important role here too: if your lab relies on FOSSS for data analysis, consider allocating grant funds to support a scientific software developer, making FOSSS contribution part of the job description for your next postdoc or research scientist hire, or planning that your graduate students will need either a longer duration of support or fewer research output expectations (or both) in order to develop the necessary competencies to both use and contribute to the tools they rely on.

Importantly, it should not only be the developer’s responsibility to prove the value of their contributions; the burden should lie with promotion and tenure committees to become familiar with how to evaluate the scope and import of software contributions (fortunately, the Research Software Alliance and Research Data Alliance are working on policy recommendations on this topic,² and CURIOSS is working to support and improve open-source program offices across academia³). Failing that, promotion and tenure committees should, at minimum, allow, request, and encourage scientists who develop software to contextualize the scope and impact of their software work (Hafer & Kirkpatrick, 2009). We, however, want to caution against the development of new and too simplistic metrics for the measure of open-source work in academia, as measuring software impact is difficult and takes time (Afiaz et al., 2023). Furthermore, Goodhart’s law postulates that every measure that becomes a target becomes a bad measure (Afiaz et al., 2023; Goodhart, 1984), as has happened to the h-index (e.g., Bartneck & Kokkelmans, 2011; Purvis, 2006; Seeber et al., 2019; Zhivotovsky & Krutovsky, 2008).

Lastly, we advocate for programs and departments to incorporate better training of students and PhD candidates in software development and maintenance (Carver et al., 2022; Guest & Forbes, 2023; Millman & Pérez, 2018): this would not only increase the software engineering skills of academics generally (Connolly et al., 2023), but would also send a strong signal about the importance of a well-written, reusable code to junior researchers. While training programs such as, for example, Software Carpentry⁴, CodeRefinery⁵, INTERSECT⁶, Neurohackademy⁷, URSSI⁸, or Neuromatch Academy⁹, give excellent training courses and summer schools, we also call on teaching coordinators of universities to reflect an increasing need for programming literacy in their neuroscience programs. Together with the recognition for writing open-source software, this would be an important step toward a healthy software ecosystem in neuroscience.

We predict that all action points discussed above will have a positive impact on the inclusivity of open-source work in academia. Changing the incentive structure such that open-source work does not have to be a privilege anymore but gets seen as what it is, a valid and critical contribution to science, will facilitate the participation of underrepresented groups. Beyond this, we strongly advocate that open-source projects in neuroscience make sure to be welcoming to everyone and to prevent any harassment, for example, by stating and adhering to Community Guidelines.

Summarizing the key points of this paper, we hope to raise awareness within the neuroscientific community about its dependence on a rather brittle structure. Your open-source software ecosystem needs your help! Immediate action can be taken by citing current software versions instead of the seminal software-describing paper, by making space for your trainees to engage in FOSSS communities, and by rewarding them (or at least not penalizing them) when they do. Beyond this, the incentive structure in academia and the policies that support it (including those created by research-performing organizations, funders, publishers, etc.) urgently need to be re-thought (Hostler, 2023; Jensen & Katz, 2023; Merow et al., 2023; Millman & Pérez, 2018; Munafò et al., 2017; Neylon et al., 2012)—not only for the sake of the academic open-source ecosystem, but for the good of the neuroscience community as a whole.

This paper does not rely on any custom code. All resources used for the Truck Factor analysis are reported in Box 1.

Conceptualization: B.U.W. Formal analysis: B.U.W. & D.R.M. Writing—original draft: B.U.W., D.R.M., & E.L. Writing—review & editing: B.U.W., D.R.M., E.L., A.G., D.S.K., A.M.S., A.D., V.L., S.M., R.O., J.M.S., & T.M.T.

The authors declare no competing interests.

The authors thank Sylvain Baillet for valuable discussion and comments on a previous version of this manuscript. This project has been made possible in part by grant number 2021-237679 from the Chan Zuckerberg Initiative DAF, an advised fund of Silicon Valley Community Foundation. V.L. and T.M.T. were supported by the Wellcome Centre for Human Neuroimaging, funded by Wellcome [203147/Z/16/Z]. T.M.T. is funded by a fellowship from Epilepsy Research UK and Young Epilepsy (FY2101).