Interdisciplinary research in artificial intelligence: Lessons from COVID-19

https://www.webofscience.com/api/gateway/wos/peer-review/10.1162/qss_a_00329

Interdisciplinarity has become a buzzword in science policy, and with good reason. Disciplines have for decades—in some cases, centuries—facilitated scientific progress by providing scholars with the scaffolding of a coherent paradigm and the possibility of standing on the shoulders of their predecessors. However, disciplinary boundaries have often proved to be a stumbling block to innovation, as growing specialization makes it ever harder (though ever more necessary) to venture into unexplored research territories and combine intellectual tools originating from different traditions (Jones, 2009). These entrenched boundaries are especially problematic when facing unprecedented research challenges that require fresh thinking and unrestrained experimentation. Such a situation presented itself recently with the outbreak of the COVID-19 pandemic. The urgency and gravity of the situation prompted researchers in epidemiology and medical science not only to mobilize all the resources available within their disciplines, but to look beyond them for new ideas and external collaborations. Among them, the alliance with artificial intelligence (AI) emerged as one of the most promising (Figure 1).

Although AI techniques have a long history, the field has recently been revived by the escalating power of computational technologies and the growing availability of data on social and natural phenomena. This has led to the development of new machine learning (ML) approaches, which have yielded remarkable results within and beyond data science (Cardon, Cointet et al., 2018; Frank, Wang et al., 2019). Recent studies have shown that AI/ML techniques are indeed changing the “way of doing science,” from agenda-setting and hypothesis formulation to experimentation, knowledge sharing, and public involvement, with a considerable impact on scientific practices (Agrawal, McHale, & Oettl, 2019; Bianchini, Müller, & Pelletier, 2022; Birhane, Kasirzadeh et al., 2023; Cockburn, Henderson, & Stern, 2019; Koehler & Sauermann, 2024; Van Noorden & Perkel, 2023; Xu, Liu et al., 2021)¹.

The coronavirus pandemic hits at the peak of this cycle of AI hype and, unsurprisingly, many scholars quickly embraced the idea of adopting AI techniques to tackle the challenges presented by COVID-19 (DeGrave, Janizek, & Lee, 2021; Khan, Mehran et al., 2021; Roberts, Driggs et al., 2021)². Opportunities for collaborative funding have emerged globally to bring various scientific communities together, and researchers from different backgrounds have come together to try to harness the potential of AI in COVID-19 research (Ahuja, Reddy, & Marques, 2020; Luengo-Oroz, Hoffmann Pham et al., 2020). Some of these collaborations offered substantial contributions to the fight against the pandemic. By reviewing some of the most cited papers in our data set, we found some interesting uses of AI for COVID-19 research, particularly for making sense of large literature or data archives (e.g., Mistry, Chuguransky et al., 2021; Salari, Hosseinian-Far et al., 2020; Wynants, Van Calster et al., 2020), and some thoughtful assessments of the effectiveness of AI and big data in medicine (e.g., Agbehadji, Awuzie et al., 2020; Wang, Ng, & Brook, 2020a). Many other publications at the AI/COVID-19 intersection, however, never gained much visibility or scientific traction. What can explain these contrasting outcomes?

Previous research shows that (large) interdisciplinary teams produce more cited research and high-impact papers (Fortunato, Bergstrom et al., 2018; Wuchty, Jones, & Uzzi, 2007), and that diversity—not only epistemic, but also institutional and ethnic—is beneficial for producing novel, valuable ideas (Taylor & Greve, 2006). Teams comprising researchers with different backgrounds, methodological approaches, and experience have access to a broader pool of knowledge, which allows them to produce more creative outputs than those produced by less collaborative science (Gargiulo, Castaldo et al., 2022; Stephan, 2012; Uzzi, Mukherjee et al., 2013). This can be explained by the functional diversity of teams, that is, differences in the way scientists encode problems and attempt to solve them; as Hong and Page (2004) put it succinctly: “diversity trumps ability.” Collaborative projects also serve to boost visibility by exposing scientific findings to a wider and more diverse readership (Leahey, 2016). In the case of COVID-19 research, this suggests that collaborations between AI experts and clinicians may result in successful research outcomes, as domain specialists could provide their “on-the-ground” knowledge to identify promising areas for investigation, while technology experts could offer access to the latest computational methods.

Team diversity, however, is not without its disadvantages. Teams that are too large and heterogeneous often suffer from lower consensus-building, cognitive diversity, higher coordination costs, and emotional conflict. As diversity increases, it may become more difficult to convert specialized expertise into scientific outputs (Lee, Walsh, & Wang, 2015). Studies show that team performances depend more on how the team interacts than on the characteristics of its members (Woolley, Chabris et al., 2010), and that most successful collaborations seem to be achieved through efforts that, while interdisciplinary, combine relatively close fields (Yegros-Yegros, Rafols, & D’Este, 2015)³. Furthermore, some physicians were and continue to be suspicious of AI, for it threatens their professional expertise and challenges their authority over clinical truth and, more generally, the place of medicine within society (see Hanemaayer (2021) for a discussion on the historical resistance to AI and algorithmic computer technologies in medicine). Recent empirical evidence suggests indeed that it is difficult for AI-engaged research to mix well with more “traditional” domain-specific research (Duede, Dolan et al., 2024). Consequently, difficulties may have arisen in collaborations between AI and COVID-19 experts, potentially leading to less impactful and visible scientific outcomes compared to teams composed solely of AI or clinical specialists⁴.

In this paper, we examine the impact of interdisciplinarity by investigating a large corpus of scientific publications at the intersection of COVID-19 and AI (about 15,000 papers retrieved from the COVID-19 Open Research Dataset, CORD-19 (version 2021-08-09) and supplemented by other metadata from Altmetric and OpenAlex), and studying which forms of interdisciplinarity are more strongly associated with scientific impact. In the remainder, we first describe the metrics of interdisciplinarity used in our study, and then link these metrics to three indicators of scientific “success,” namely the number of citations, online visibility, and outreach to other disciplines.

Our analysis combines data from three different databases—CORD-19, OpenAlex, and Altmetric—and is based on the preprocessing protocol illustrated in the Supplementary material (Figure S1). The COVID-19 Open Research Dataset (CORD-19) is a growing corpus of publications on COVID-19 and other coronavirus infections (Wang, Lo et al., 2020b). It includes, in the period that we considered (from December 1, 2019 to August 31, 2021), around 600,000 documents from different sources, including WHO, PubMed central, bioRxiv and medRxiv. Within this large corpus, we focused specifically on a subset of publications that included, in their abstract or title, at least one keyword related to AI. Our list of around 300 AI keywords (see Table S1 in the Supplementary material) was created by merging the terms mentioned in the Wikipedia AI Glossary for AI with other “AI vocabularies” (Baruffaldi, van Beuzekom et al., 2020; Bianchini et al., 2022; Gargiulo, Fontaine et al., 2023).

For each paper in this subset, we retrieved additional metadata from OpenAlex. We discarded all documents with missing information and obtained a final corpus of 16,148 AI publications on COVID-19 (COVID-19+AI data set). We retrieved the metadata for all the references cited by the publications in our corpus (approximately 300,000 unique papers) and for all the papers that cite them (approximately 200,000 papers). OpenAlex metadata included the DOI, which we used for retrieving the “attention score” for each paper in the COVID-19+AI data set from the website Altmetric.com. The score provides a measure of online visibility for scholarly contents (e.g., mentions on the news, in blogs, and on Twitter; article page-views and downloads; GitHub repository watchers). Finally, we used the author identifier in OpenAlex to retrieve the previous publications of all 87,552 authors present in our corpus (around 150,000 papers) and the institutions to which they are affiliated.

The concept of interdisciplinarity is multifaceted and often ambiguous, and there is no consensus on the definition and operationalization of interdisciplinary research (Fontana, Iori et al., 2020, 2022; Huutoniemi, Klein et al., 2010; Leydesdorff & Rafols, 2011; Porter, Cohen et al., 2007; Wang & Schneider, 2020; Yegros-Yegros et al., 2015). Here, we use different measures of interdisciplinarity that consider the diversity of team members and references cited in a paper.

Each document, i, in our data is characterized by a set of authors (A_i), a set of references and citations (R_i, C_i), a set of AI keywords, if any, (W_i), the journal where it is published (J_i), and its altmetric score (M_i). Each author, a, in our corpus is associated with their list of papers (P_a) and with their three most recent papers (⁠$Pa3$⁠).

Using this notion of distance, we define two types of interdisciplinarity metrics. The first is related to team composition (measuring the disciplinary span of the previous papers by the contributors of a paper); the second is related to the knowledge mobilized in the paper (measuring the disciplinary span in papers’ references). For each dimension (team and knowledge), we introduce a further distinction between interdisciplinarity metrics specifically related to AI, and the more general interdisciplinarity, providing us with four main different metrics:

• AI Team Expertise is the fraction of previous AI publications for each author, averaged over the entire team:

  $AITeamExpertisei=1#Ai∑a∈Ai#j∈PaWj≠#Pa$
• Share AI References is the fraction of cited references related to AI:

  $ShareAIReferencesi=#j∈ℛiWj≠#ℛi$
• PMI (Team) is the average disciplinary dispersion (in term of journal distances) of team authors:

  $PMITeami=1#Ai∑a∈Ai13∑k≠l∈Pa3DJkJl$
• PMI (References) is the average distance among all the journals cited in the references:

  $PMIReferencesi=1#ℛi×ℛi∑uv∈ℛi×ℛiDJuJv$

The first two metrics measure the share of AI in the author teams and knowledge mobilized by the publications, respectively. The last two measure levels of general interdisciplinarity in the teams and the knowledge mobilized by the publications.

Thus, for the sake of completeness, we also calculate the Rao-Stirling disparity, D_i, for teams and knowledge composition. Our metric, as well as the Rao-Stirling disparity, takes into account the relative distance among the journals/disciplines, avoiding treating two journals/disciplines that are very similar in content as truly different.

We also extend the analysis to two other dimensions traditionally used to define interdisciplinarity: variety and balance. The variety, V_i, is the count of the number of different journals where the authors previously published (for team) and of the different journals cited in the references (for knowledge). The balance, B_i, is the Gini index of the frequency associated to each journal for authors previous publications (teams) and for the references (knowledge).

Descriptive statistics of the variables used for this study are reported in Table S2 in the Supplementary material.

By running an LDA (Latent Dirichlet Allocation) topic modeling on the abstracts of the papers in our corpus, we obtained five distinct areas in which AI/ML techniques have been applied (Figure 2):

• Societal Issues (including epidemiology and infodemics), with some recurrent terms such as social medium, infectious disease, mental health, reproduction number, social distance, etc.;

• Medical Imaging: chest X-ray, chest scan, tomography, etc.;

• Diagnosis and Prognosis: clinical trials, risk factors, mechanical ventilation, etc.;

• Treatments and Vaccines: molecular docking, spike protein, gene expression, drug discovery, etc.;

• Public Health: public health, contact tracing, health system, face mask, etc.

A closer reading of the terms characterizing each topic suggests that AI has found a multitude of applications (Bullock, Luccioni et al., 2020; Naudé, 2021; Piccialli, di Cola et al., 2021; Yang, Nelson et al., 2020)—see Table S3 in the Supplementary material. In the case of societal issues, AI seems to have been used mainly for predicting the spread of disease over time and space, modeling public policy interventions (e.g., social distancing) and risk assessment, and fighting misinformation and disinformation on social media. In the case of medical imaging, we see the deployment of deep learning models (e.g., CNN) to detect signs of COVID-19 from X-ray images and computed tomography (CT) scans. Another area of application, particularly of machine learning and deep learning, is the identification of possible treatments and vaccines, as well as the repurposing of existing drugs. Finally, AI appears to support the management of the public health system, for example, robotics providing assistance in the delivery of healthcare tasks.

Each application area may have required specific skills and know-how from researchers with diverse backgrounds and experiences, as well as the (re)combination of different types of knowledge. Our corpus reveals a high level of general interdisciplinarity both in the teams and in the knowledge mobilized by the publications across all research topics—with a slightly higher knowledge heterogeneity in societal issues and diagnosis/prognosis (Figure 3, top).

In the case of AI, we observe very different scenarios at the topic level. Indeed, the share of teams with more AI experts is markedly higher in medical imaging and public health research, whereas teams working on vaccines, treatments, and prognosis seem to rely very little on AI knowledge (Figure 3, bottom).

We model the various impact measures (i.e., the number of citations received by the publication, the Altmetric attention score, and the interdisciplinary spread) as a function of the different interdisciplinarity metrics discussed earlier and a set of control variables, namely: AI Collaborator (=1 if the team includes at least one AI researcher); Top AI Collaborator (=1 if the team includes an AI researcher with past number of citations in the top decile of the citation distribution); Academic Age (average academic age of team members, in logs); Past Impact (average h-index of team members based on past publications, in logs); Nb. Countries (number of participating countries within a team, in logs); and Nb. References (number of cited references, in logs). We also included a complete set of fixed effects for the month of publication and the dominant topic. The number of citations is a count variable and was modeled using a negative binomial regression. The continuous variables—attention score and interdisciplinarity spread—were modeled using ordinary least square regressions.

As shown in Tables 1 and 2, the most notable result to emerge from our model is that collaborations with researchers experienced in AI (AI Collaborator) do not have a significant impact, and those involving a high share of researchers with established track records of AI publications (AI Team Expertise) receive, ceteris paribus, fewer citations, have less online visibility, and struggle to reach distant disciplines. Only those teams that include a top AI researcher (Top AI Collaborator) present a positive impact on citations received by their publication, albeit this impact is not strong. Similarly, the ratio of AI-related references (Share AI References) has a null or negative impact on the Altmetric attention score and interdisciplinary spread. All in all, research interdisciplinarity limited to AI does not seem to have any influence on the impact of COVID-19 publications, and when it does, this influence is negative.

What appears to ensure the impact of a publication is, above all else, the interdisciplinarity of the knowledge mobilized via its references, that is the actual epistemological diversity of the research conducted by a team. Regardless of how we operationalize this diversity, we find a systematic positive effect on all impact measures (except for disparity in the models on the number of citations and attention score). The effect is consistently higher than that of more classic features, such as past impact or the number of affiliated countries. The overall diversity of team members generally has a much less strong, significant and in many cases negative effect.

We made sure that our main findings are robust to alternative specifications and arbitrary choices. First, we performed the same modeling exercise using OpenAlex “concepts” instead of journals. We replicated the models by considering level-0 concepts (e.g., computer science) associated with each journal and level-1 concepts (e.g., machine learning), which are more granular. A journal can be associated with more than one concept; in this case, we considered the concept with the highest “confidence score” provided by OpenAlex. Second, we considered the five most recent papers by each author instead of three. Third, we re-estimated the models for the number of citations with a quasi-Poisson instead of a negative binomial regression. Finally, we excluded all publications that were still preprint as of December 31, 2023.

The COVID-19 pandemic sparked a global research effort to address this unprecedented event. The scientific system responded promptly to the early stages of the virus and the international scientific community called upon its diverse expertise to assess the clinical and pathogenic characteristics of the disease and to formulate therapeutic and epidemiological strategies to cope with it. Policymakers were also quick to seek advice from ethicists, sociologists, and economists on how best to deal with the crisis (Chahrour, Assi et al., 2020; Fry, Cai et al., 2020). Against this backdrop, AI applications represented a promising approach to facing many of the challenges posed by the pandemic. Yet a number of studies focusing on the application of AI-based approach to COVID-19 research have identified various barriers and shortcomings. They include poor data quality and flow, as well as the lack of global standards and database interoperability (e.g., genetic sequences, protein structures, medical imagery and epidemiological data); the inability of algorithms to work without sufficient knowledge of the domain; overly exacting computational, architectural, and infrastructural requirements; and the legal and ethical opacity associated with privacy and intellectual property (Bullock et al., 2020; Khan et al., 2021; Luengo-Oroz et al., 2020; Naudé, 2020; Piccialli et al., 2021).

In this paper, we have analyzed the role played by different forms of interdisciplinarity, both at the team level and in the research conducted, and their repercussions on various measures of scientific impact. Our research was, in part, motivated by the fact that policy initiatives around the world have emerged—and continue to emerge—aimed at encouraging collaboration between the AI community and specialists in various domains. However, we have no direct evidence of the effectiveness of these initiatives.

Our study provides an important takeaway message for policy makers and science administrators: Collaborations involving AI researchers did not necessarily result in more impactful science. As our analysis revealed, the visibility, relevance, and spread of the publications we considered all seem to be linked to the diversity of references rather than that of authors. What generates high-impact science, in other words, is not the possible interdisciplinarity associated with team diversity, but the actual epistemological diversity hardwired into a paper.

We are grateful to the handling editor and the two referees for their careful reading and constructive suggestions that significantly improved the earliest versions of our manuscript. We also thank Cinzia Dario, André Lorentz, Björn Jindra, Patrick Llerena, and Lili Wang for their helpful comments. Any opinions and conclusions expressed herein are those of the authors.

Diletta Abbonato: Conceptualization, Data curation, Formal analysis, Methodology, Visualization, Writing—original draft, Writing—review & editing. Stefano Bianchini: Conceptualization, Funding acquisition, Methodology, Supervision, Writing—original draft, Writing—review & editing. Floriana Gargiulo: Conceptualization, Funding acquisition, Methodology, Visualization, Writing—original draft, Writing—review & editing. Tommaso Venturini: Conceptualization, Funding acquisition, Methodology, Visualization, Writing—original draft, Writing—review & editing.

The authors have no competing interests.

This work was supported by the La Mission pour les Initiatives Transverses et Interdisciplinaires (MITI) of the Centre National de la Recherche Scientifique (CNRS). This work was also supported by the European Union—Horizon 2020 Program under the scheme “INFRAIA-01-2018-2019—Integrating Activities for Advanced Communities”, Grant Agreement n.871042, “SoBigData++: European Integrated Infrastructure for Social Mining and Big Data Analytics” (https://www.sobigdata.eu). Stefano Bianchini received financial support through the SEED project—Grant agreement ANR-22-CE26-0013. Floriana Gargiulo received financial support through the ScientIA project ANR-21-CE38-0020.

Data and code to reproduce the main analyses and robustness tests are available here: https://github.com/zabbonat/AI-COVID19-interdisciplinarity.