Exploring evidence selection with the inclusion network

https://www.webofscience.com/api/gateway/wos/peer-review/10.1162/qss_a_00287

Systematic reviews have been widely adopted in a number of fields in the social sciences, ecological sciences, software engineering, and health sciences, in order to synthesize scholarly literature and provide guidance for practice. To limit bias and ensure the validity of their conclusions, systematic reviews typically use relatively standardized procedures (Cooper, Hedges, & Valentine, 2019; Haddaway, Macura et al., 2018; Kitchenham, Budgen, & Brereton, 2015; Okoli, 2015; Page, McKenzie et al., 2021a), which have been customized and refined over decades in multiple fields (Chalmers, Hedges, & Cooper, 2002; Gurevitch, Koricheva et al., 2018). Despite these attempts at standardization and reducing bias, it is common for systematic reviews published at the same time on the same topic to yield different conclusions (Gøtzsche, 1994; Jadad, Cook, & Browman, 1997; Papatheodorou & Evangelou, 2022), which can pose challenges for practical application.

One crucial step in a systematic review is the selection of evidence. This consists of defining the selection principles (known as the “inclusion criteria”) and exhaustively identifying relevant literature on a given topic¹. This search and screening process is often specified in advance in a protocol (Page, Shamseer, & Tricco, 2018) and typically uses a comprehensive search in multiple databases, with well-defined and consistently applied evidence selection criteria (Lefebvre, Glanville et al., 2022; Rethlefsen, Kirtley et al., 2021). Evidence selection results in a list of publications, called the included primary study reports, which form a special subset of a systematic review’s citations. However, this step is known to admit variation: Even methodologically rigorous reviews that address the same question can be discordant in their evidence selection (Créquit, Trinquart et al., 2016; Trinquart, Johns, & Galea, 2016). In particular, discordance in evidence selection is frequently observed in conflicting systematic reviews (Bolland & Grey, 2014; Coarasa, Das et al., 2017; Useem, Brennan et al., 2015).

Network analysis has been used in the task of assessing the reliability of scientific knowledge (Duyx, Urlings et al., 2017a; Greenberg, 2009, 2011; Leng, 2018; Shwed & Bearman, 2010). The purpose of this paper is to design and test a new approach that uses network analysis to assess risks in a systematic review that decision-makers want to use as the foundation of their action. We define risk as any information concerning a systematic review that can reduce a user’s willingness to use the systematic review as the foundation of their action. In this paper, we aim to answer two research questions:

The remainder of the paper is organized as follows. Section 2 presents background, including reliability issues in systematic reviews and the use of network analysis to assess the reliability of scientific knowledge. Section 3 answers RQ1 by detailing the design of our new approach. Sections 4 and 5 answer RQ2. In particular, Section 4 introduces our two case studies and how we implemented our approach on the two case studies, and Section 5 presents the discoveries we made from the two case studies. We discuss the difference between our approach and related work, especially claim-specific citation network analysis (Greenberg, 2009, 2011), as well as our limitations and future work in Section 7. We conclude with our primary findings in Section 6.

Our work is built upon two threads of prior research: first, reliability issues in systematic reviews, and second, the use of network analysis to assess the reliability of scientific knowledge.

Systematic reviews are a methodology developed to ensure that scientific evidence can be accumulated scientifically and summarize the state of knowledge on a particular issue (Chalmers et al., 2002). The validity of review conclusions has long concerned researchers involved in methodological development. As early as 1982, Harris Cooper, one of the founders of modern research synthesis, conducted a comprehensive analysis of threats to validity in the systematic review process. He divided the review process into five stages: (a) problem formulation; (b) data collection; (c) evaluation of data points; (d) data analysis and interpretation; and (e) presentation of results. The evidence selection process is influenced by stage a and spans stages b and c. Within these three stages, Cooper (1982) identified the following threats to validity: a lack of consensus on the meaning of a concept, too narrow a definition of a concept, a lack of attention to how studies operationalized a concept, failure to locate all studies pertinent to the topic of interest, a mismatch between the target population and the population covered in the studies collected, improper evaluative criteria (e.g., confirmatory bias), and unreliable primary research (the only threat that is out of reviewers’ control, as Cooper (1982) pointed out).

Later, as conflicting systematic reviews emerged and started to concern researchers, Felson (1992) formulated a framework to describe threats to validity using the language of bias. In this context, a bias is a factor that can systematically drive conclusions away from the truth. Felson (1992) also proposed a taxonomy of bias. For each type of bias, Felson described corresponding methods and principles that can help eliminate or alleviate it.

Conflicting conclusions in systematic reviews have been an ongoing subject of study even in recent decades (Bolland & Grey, 2014; Coarasa et al., 2017; Hacke & Nunan, 2020; Khamis, El Moheb et al., 2019; Lucenteforte, Moja et al., 2015; Osnabrugge, Head et al., 2015; Useem et al., 2015). The overwhelming number of systematic reviews (Bastian, Glasziou, & Chalmers, 2010) and their production and selective publication for marketing purposes (Ioannidis, 2016) intensifies the importance of studying conflicting conclusions in systematic reviews.

Evidence selection discordance is frequently observed in conflicting systematic reviews. Two systematic reviews on healthcare in low- and middle-income countries searched for evidence in the same month, but came to different conclusions; the studies reviewed hardly overlapped: “There were only 16 citations in common although one reviewed 102 studies and the other reviewed 80 studies” (Coarasa et al., 2017). Meanwhile Useem et al. (2015) studied 40 matched pairs of systematic reviews (matching each review by well-known review producer Cochrane with a review by a non-Cochrane review group) that were published within 5 years of one another. They found substantial differences in primary study inclusion, and the publication sequence—evidence published subsequent to one review and prior to another review—was unable to explain discordance in 47% of the pairs of reviews examined. In another study comparing Cochrane and non-Cochrane matched pairs, Hacke and Nunan (2020) found only one out of 24 matched pairs included the same studies. For two-thirds of the matched pairs, they could not explain the evidence inclusion discrepancies. Bolland and Grey (2014) analyzed a sample of seven meta-analyses reporting the relationship between vitamin D and bone fracture published in high-ranking general medical journals. They identified a set of trials deemed eligible for the meta-analyses. Of the seven meta-analyses, Bolland and Grey (2014) found that four reviews included all eligible trials; the remaining three missed three to eight trials. Siontis, Hernandez-Boussard, and Ioannidis (2013) discovered that even when meta-analyses stated similar eligibility criteria, they still selected different evidence, although in this case, the reviews Sointis et al. studied did not reach conflicting conclusions.

Recently, attention has turned to reproducing systematic reviews. Low, Ross et al. (2017) investigated whether two independent systematic review teams provided with identical objectives, clinical data, resources, and time would yield the same results when reviewing industry data. The two systematic review teams came to broadly the same findings, with differences regarding how the treatment studied impacted the risk of cancer and the benefits in spinal fusion surgery. Their methods, results, and interpretations still differed slightly. Low et al. (2017) took a positive view of the variations in different meta-analyses, because closed data with a single interpretation of data may “lead people to believe there is no other possible approach.” Concerns regarding the reproducibility and replicability of evidence synthesis led to a community effort called the REProducibility and Replicability In Syntheses of Evidence (REPRISE) project, which plans to crowdsource reviewers in order to replicate the search and inclusion of evidence from a sample of published systematic reviews (Page, Moher et al., 2021b).

Current approaches for studying discordance developed by the systematic review community primarily rely on content analysis (Bolland & Grey, 2014; Hacke & Nunan, 2020; Katrak, Bialocerkowski et al., 2004; Osnabrugge et al., 2015) combined with methods referred to as “critical appraisal” or “critical assessment” of systematic reviews (Katrak et al., 2004; Osnabrugge et al., 2015). The conclusions of such studies report discordance in evidence selection as the number or fraction of overlapping included studies and number or fraction of differences. Occasionally, data analytics and visualization have been used. Lucenteforte et al. (2015) used colored circles (“traffic light visualizations”) to visually compare pairs of heart attack treatments by clustering 36 systematic reviews; colors indicate superiority (green), nonsuperiority (yellow), and discordance (red). Créquit et al. (2016)’s Figure 4 used network visualization to show the percentage of treatment comparisons covered by systematic reviews in each year from 2009 to 2015, from 77 randomized controlled trials of lung cancer treatments; nodes are treatments and edges are treatment comparisons. The only visual and data analytical approach used in studying evidence inclusion we are aware of is a tool called GROOVE (short for Graphical Representation of Overlap for OVErview), which provides spreadsheet templates for authors developing overviews of multiple systematic reviews (Pérez-Bracchiglione, Meza et al., 2022). GROOVE’s focus is on assessing the overlap of the primary studies (evidence inclusion) to derive appropriate weights for statistical summaries in the overviews rather than understanding evidence selection discordance.

Greenberg (2009, 2011) studied scientific belief systems, focusing on the impact of uneven citation of nonconcordant evidence. He introduced the concept of a claim-specific citation network, “the set of all papers (nodes) containing statements regarding a specific claim or related set of claims and the citations (arrows; directed edges) from these statements to other papers” to represent the scientific belief system (Greenberg, 2011). Using his claim-specific citation network construct, Greenberg traced the factual foundation of a widely held belief important to the care of patients with a muscle disorder called inclusion body myositis and concluded that the belief was unfounded. He identified citation bias (“When relevant data addressing a claim are not cited” (Greenberg, 2011)) and the amplification or the “lens effect” by a few influential papers “containing no data on claim validity” (Greenberg, 2009).

Claim-specific citation networks were later adapted² by Leng (2018) to reveal research underutilization, biased research utilization, and their impact on the conclusions drawn by narrative scientific reviews. Leng (2018) constructed a claim-specific citation network of reviews centered on the four pre-1984 randomized controlled trials (RCTs) investigating whether fat-controlled diets were effective for preventing coronary heart diseases. Forty-five percent of the reviews were supportive, with the remainder evenly split between neutral and unsupportive. Consideration of more evidence was associated with a neutral position on the benefit of fat-controlled diets, and many reviews only cited the one RCT supportive of a fat-controlled diet while ignoring the three RCTs unsupportive of this claim. The four RCTs Leng collected constituted a tiny evidence set that reviewers had no excuse to miss, and all RCTs should have been utilized, because Leng selected only reviews published at least 2 years after the last RCT report was published. Yet reviews made fewer than half the possible citations to RCTs: In the network as a whole, Leng found evidence utilization of only 49%. By contrast, the neutral reviews had higher evidence utilization (68%). A Pearson’s chi square test confirmed citation bias (Leng, 2018).

Trinquart et al. (2016) used an exponential random graph model (Robins, Pattison et al., 2007) to test citation homophily based on the stance a report took towards a scientific claim (supportive, against, and inconclusive) and found that publications tend to cite other publications with the same stance. Their work further consolidated the association between citation bias and unreliable knowledge: When papers on each side only cited papers of the same stance, the literature left readers little impression of an ongoing scientific controversy. Duyx, Urlings, Swaen, Bouter, and Zeegers published a series of empirical studies on determinants of citation (Duyx et al., 2017a, Duyx, Urlings et al., 2017b, 2019, 2020; Urlings, Duyx et al., 2019a, 2019b, 2020, 2021). Most importantly, they contributed a universal method for detecting citation bias with statistical rigor. Their method can identify publication characteristics that are associated with citation frequency by modeling the relationship between any chosen characteristic of the publication (e.g., conclusion, author affiliation, sample size) and the probability of a potential citation path being realized (Urlings et al., 2021).

Knowing whether a controversy has reached its conclusion can help one assess the risk of adopting a particular knowledge claim. To characterize the level of consensus in a network, Shwed and Bearman (2010) measured network cohesion using the modularity score (Newman, 2018), scaled by the logarithm of the size of the network. They also introduced a dynamic window approach to subset a citation network temporally to reflect the scientific belief system at a given time; for a given year, the window width is determined by the median age of publications cited in a given year (where age is publication year of the citing paper minus publication year of each cited paper). Shwed and Bearman argued that their dynamic window approach was superior to the cross-sectional or fixed-window approach because the window width was theoretically justified and sensitive to the level of scientific activity. It was also shown to be more sensitive to shifts in consensus. By combining the scaled modularity score and dynamic window approach, Shwed and Bearman were able to measure the change in consensus over time with higher sensitivity. They found that changes in scaled modularity score using their dynamic window approach tracked their expectations for studies that were (a) never contested (noncarcinogenicity of coffee), (b) once contentious but currently consensual (carcinogenicity of smoking, climate change), and (c) currently contested (carcinogenicity of cell phones, relationship between vaccination and autism).

Controversy mapping is a “conceptual and methodological toolbox” for studying scientific controversy, and it incorporates network analysis (Venturini & Munk, 2022). However, controversy mapping differs from the methods introduced so far in at least three aspects: (striving for) a more sympathetic attitude towards all players³, resistance to a (hasty) reduction of a controversy, and a refusal to specify their approach in a conventional way⁴ (Venturini, 2010). Although not open to those uninitiated in its use, controversy mapping is extremely powerful and adaptive to navigate epistemic communities at clash with each other, as Jacomy (2020) exemplified in his analysis of the controversy of scale-freeness in the field of network science.

Reliability issues in systematic reviews make it important to assess systematic reviews before they are used for decision making. Network analysis has shown its strength in assessing the reliability of scientific knowledge, primarily through approaches based on claim-specific citation network analysis. The methodological insight of claim-specific citation network analysis is that a claim-specific citation network is “a published record of a belief system” and the propagation from data to belief was documented in it (Greenberg, 2011). In this case, the entire network is about one claim or one piece of scientific knowledge. Our work joins these two threads of research with an important alteration: We treat a given systematic review (the target) as a documentary representation of some scientific knowledge, and we assess the risk of adopting its knowledge by looking at how the given systematic review’s chosen evidence is compared to evidence chosen by other systematic reviews studying similar topics (the companion reviews).

The purpose of this paper is to design and test a new approach that uses network analysis to assess risks in a systematic review that decision-makers want to use as the foundation of their action. We define risk as any information concerning a systematic review that can reduce a user’s willingness to use the systematic review as the foundation of their action. The definition also forms our first design principle that users, based on their own knowledge and values, make the final judgment of whether they find a risk.

Based on what we reviewed in Section 2.1, prior research (e.g., Cooper (1982) and Felson (1992)) has laid out many threats to the validity of review conclusions. Yet in addition to those threats to validity, we are aware that some risks can be unexpected, clear only on retrospection or through the participation of people with specific knowledge related to an issue. With this in mind, the second design principle is that the approach must have a component that enables users to discover unknown risks.

We now come to the third design principle, the principle that differentiates our approach from claim-specific citation network analysis (introduced above in Section 2.2). This principle is also specific to the study of evidence selection: We treat a systematic review as a documentary representation of some scientific knowledge, and we assess the risks of adopting the knowledge by looking at how the systematic review’s chosen evidence compares to evidence chosen by other systematic reviews studying similar topics. This principle incorporates the social and historical aspect of scientific knowledge, as the standards for valid scientific knowledge vary across different scientific communities and evolve with time (Cetina, 1999; Kuhn, 2012). Thus, one way, and perhaps the only way, to discover risks associated with community differences and temporal evolution in evidence selection, is by comparing one systematic review to other systematic reviews researching similar research topics.

Network analysis cannot discover all possible risks under our definition. Further considerations thus include whether network analysis techniques have the analytical affordance to detect a risk and whether network analysis can justify its superiority in detecting that risk over other methods. For example, a mismatch between the target population and the population covered in the studies collected (Cooper, 1982) can be detected by reading a systematic review’s inclusion criteria, or even the title. We focus on evidence selection in this paper for two reasons. First, evidence selection discordance is a frequent observation in conflicting systematic reviews and a common concern (see Section 2.2). Second, network analysis allowed us to vividly see evidence selection discordance in our pilot study (Hsiao, Fu, & Schneider, 2020).

Therefore, the fourth design principle is that appropriate risks to be assessed by this approach are most likely those that can reveal themselves through quantification by network metrics but not by other means.

We document all four design principles in Table 1 for easy reference.

We propose a network construct called the inclusion network. The inclusion network is a bipartite network with two types of nodes: One represents systematic review reports (SRRs), and the other represents primary study reports (PSRs). A PSR is “included” in an SRR if it is used in that SRR’s evidence synthesis. In an inclusion network, if an SRR includes a PSR, there is a directed edge from the SRR to the PSR. Each included PSR is cited in the SRR, but not all references cited in an SRR are included in the evidence synthesis; thus the inclusion network is a subset of the paper’s citation network. Included PSRs can be determined with high confidence, as SRRs are expected to report them in their text, tables, or appendices.

The data collection process reflects Design Principle 3. To prepare their data set, users first identify a collection of SRRs (i.e., the companion SRRs) that study similar research questions to the SRR to be assessed (i.e., the target SRR). Currently, we allow users to define their collection strategy. After collecting SRRs, they extract all included PSRs from the SRRs. At a minimum, three data files should be prepared: an attribute list recording each node’s unique identifier, publication date, and type (SRR or PSR); an edge list specifying all inclusion relationships; and a file containing the dates on which each SRR completed its last search. Users can refer to Section 4.1 and the documentation of the two data sets associated with our case studies (Clarke, Lischwe Mueller et al., 2023; Fu, Hsiao et al., 2023) for more details about data preparation.

Our design of network metrics reflects Design Principle 4. We designed the adjusted Jaccard similarity as the main metric we will rely on for risk assessment. It modifies the Jaccard similarity (JS) to adjust for differences in the evidence available to be found at different points in time. The Jaccard similarity (JS)—a widely used similarity measure in graph analytics—quantifies a pair of SRRs’ similarity in choice of evidence. The adjusted JS adjusts for differences in the pool of available evidence due to the temporal sequence depicted in Figure 1. Two reviews (even SRRs published in the same month) typically searched the literature at different times and, consequently, research not yet in search indexes or published at the time of the earlier search can be included only in the SRR with the later search date. In the example shown in Figure 1, PSR 3 was published after the search date for SRR A and only has the possibility to be included in SRR B, not SRR A.

We use the individual-level average adjusted JS to assess the stability of an SRR’s evidence selection standards: whether an SRR’s choice of evidence continued to be accepted by other SRRs or the evidence it selected was no longer used by other SRRs.

x ∈ {adjusted JS(u_i, u_k) : Rank(u_k) ≤ r_(ts), i ≠ k}

The individual-level average adjusted JS is computed at each search date accounted for in the temporal sequence (t_s). The individual-level average adjusted JS for a particular SRR at t_s(y(t_s, u_i)) is defined as the average of all adjusted JS (x) of u_j with all other SRRs u_k having Rank(u_k) less than or equal to r_(ts)⁵. When computing the time series of this statistic, each SRR is time-stamped by its last search date rather than its publication date, because, again, the search date is more consequential to the resulting PSR list than the publication date, and there is often considerable delay between last search and its publication. This statistic, the individual-level average adjusted JS, has a particular property that can help detect changes in evidence selection standards: If the value computed at a given time point t(r_ts) is larger than the value computed at the previous time point t(r_ts − 1), it means that the SRR or SRRs with rank r_t are more similar to u_i in evidence selection than the SRR or SRRs with rank lower than r_ts (i.e., adding the adjusted JS between u_i and the SRR or SRRs with rank r_ts into the calculation of the average helps to bring up the average).

Our assessment process consists of three steps. Step 1 (Visualization Assessment) reflects Design Principles 1 and 2 and is exploratory and mostly analyst-driven. Mainly, we examine the entire network and the position of the target SRR in it, notice features that capture our attention, and move back and forth between the network and the content of the SRRs to identify anything we judged to be a risk.

Step 3 (Time Series Assessment) reflects Design Principle 4 and detects instability of evidence selection standards using time series based on the individual-level average adjusted Jaccard similarity. We describe the three steps of our assessment process in detail in Section 4.2.

In two case studies, we applied our inclusion network approach shown in Figure 2.

We prepared two inclusion networks, each based on data initially collected by researchers interested in studying systematic reviews addressing a given topic. This resulted in two data sets and we have made both available in the Illinois Databank: the exercise prescription (ExRx) data set (Clarke et al., 2023) and the salt controversy data set (Fu et al., 2023).

For the ExRx data set (Clarke et al., 2023), we obtained 27 SRRs⁶ investigating the relationship between physical activity and depressive symptoms by extending a previous data set from domain expert Caitlin Clarke (Clarke, 2019), which has also been used to explore syndemic approaches to exercise science intervention research (Clarke & Adamson, 2023). Our SRRs were limited to systematic reviews and meta-analyses published between 2013 and 2020 that investigated the relationship between physical activity and depressive symptoms. Exercise is often regarded as a subgroup of physical activity in the field of kinesiology/exercise science (Caspersen, Powell, & Christenson, 1985). For the rest of this paper, we generally use more colloquial terminology: “exercise” instead of “physical activity” and “depression” instead of “depressive symptoms.”⁷ A search conducted between November 2019 and February 2020 ultimately generated 27 included manuscripts. To construct the inclusion network, contributor NLM manually screened the PDFs of the 27 SRRs to identify all included PSRs (365 PSRs in total). The ExRx inclusion network contains 27 SRR nodes, 365 PSR nodes, and 589 edges.

We choose SRR #2, “Exercise for Depression” by Cooney, Dwan et al. (2013) as our target SRR. We did not strictly follow the Data Collection Process specified in Section 3.2: We did not start from a target review in our data collection but used an existing data set. SRR #2 was appropriate as a target review in our inclusion network on the relationship between physical activity and depressive symptoms due to its objective, “[t]o determine the effectiveness of exercise in the treatment of depression in adults compared with no treatment or a comparator intervention” (Cooney et al., 2013). In addition, SRR #2, a Cochrane Review by a distributed team and with at least four of its six authors outside of the field of exercise science, produced a less positive conclusion regarding the relationship between exercise and depression than the other 26 SRRs and received serious criticism from prominent researchers in the field (Clarke, 2019), making it an interesting SRR to study.

We treated the remaining 26 SRRs as companion SRRs. Although they all investigated the relationship between physical activity and depressive symptoms, their exact research questions and inclusion criteria are noticeably different⁸.

Sometimes SRRs study a facet of the research question raised by the target SRR (i.e., the relationship between exercise and depression). We name such SRRs faceting SRRs with respect to the target SRR. A facet is a certain aspect of the research question. Our target SRR, SRR #2, studies the relationship between exercise and depression, and several facets of its research question are considered in other ExRx SRRs: particular types of exercise, particular age groups, and particular levels of disease severity. For example, SRR #4 only considers yoga and depression, and SRR #25 only considers resistance exercise training and depression, making these both faceting SRRs with respect to SRR #2. Likewise, SRR #1 is limited to the elderly and SRR #24 is limited to adolescents and young adults. SRR #21 concerns a particular level of disease severity, major depression. Occasionally, facets were combined to further limit an SRR’s research question. For example, SRR #19 (Cramer, Anheyer et al., 2017) studied the “efficacy and safety of yoga interventions in treating patients with major depressive disorder.”

Faceting SRRs of the target SRR can be quite useful for assessing the target SRR using the value of adjusted JS. For example, if the adjusted JS between the target SRR (SRR #2) and an SRR that studies only yoga (e.g., SRR #4) is zero, we can infer that our target SRR has very likely overlooked evidence from yoga. This property will be used in Step 2 of our assessment process.

For the salt controversy data set (Fu et al., 2023), we obtained 14 SRRs and 68 PSRs from a claim-specific citation network collected by epidemiologists (Trinquart et al., 2016), who carried out a systematic search to identify publications addressing the effect of salt reduction on all-cause mortality and cerebro-cardiovascular diseases that were published between 1978 and 2014. The name, salt controversy, refers to the debate among scientists over the relationship between elevated salt consumption and public health risks such as stroke and cardiovascular disease (Bayer, Johns, & Galea, 2012); in particular Trinquart et al. (2016) analyzed disagreement on the top-level question (Is reducing salt beneficial for certain health outcomes?). Some of the SRRs reviewed the impact of salt reduction on other health outcomes, such as reduction in blood pressure, which fell outside of the scope of the question formulated by Trinquart et al. As a result, eight of the 14 SRRs have included PSRs that were not found in the 68 PSRs identified by Trinquart et al. (2016) (see Table S1 in the Supplementary material). Also, among the 68 PSRs, 18 were not included by any of the 14 SRRs but were identified by Trinquart et al. (2016) through their systematic search; we retained these in the inclusion network because they were technically “included” by Trinquart et al.’s search; and keeping them can help users see unused yet potentially relevant evidence. The resulting salt controversy inclusion network consists of 14 SRR nodes, 68 PSR nodes, and 184 edges.

All SRRs in the salt controversy inclusion network can be target SRRs with the exception of SRR #10, which is a faceting SRR studying a particular group: patients with systolic heart failure. We choose the target SRR from a series of Cochrane reviews⁹: SRR #5 (Taylor, Ashton et al., 2011b), #6 (Taylor, Ashton et al., 2011a), and #12 (Adler, Taylor et al., 2014). We choose the last publication of the series, SRR #12, as our target SRR and treat the remaining 11 SRRs as companion SRRs.

In Step 1 (Visualization Assessment), we created visualizations of the two inclusion networks. We examined the position of the target SRR and the entire network, noticed features that captured our attention, and moved back and forth between the network and the content of the SRRs to discover anything we judged as a risk.

In Step 2 (Quantitative Similarity Assessment), we computed regular JS, adjusted JS, and JS fold change between the target SRRs and all their companion SRRs. This step identifies evidence potentially relevant to the user that might have been overlooked by the target SRR or was published after the search date of the target SRR (referred to later simply as new evidence).

The first type is companion SRRs that have zero adjusted JS with the target SRR. If such SRRs are faceting SRRs of the target SRR, they might have examined potentially relevant evidence overlooked by the target SRR.

The second type of companion SRRs are those that finished their search after the target SR, have a high adjusted JS with the target SRR, and have a high JS fold change with the target SRR. High adjusted JS means that the target SRR and the companion SRRs share similar evidence selection criteria, and a higher JS fold change means that the companion SRR includes more new evidence (i.e., PSRs published in the interval between the target SRR’s last search date and the companion SRR’s last search date)¹⁰. By examining the PSRs included by such companion SRRs, users may be able to identify evidence that is potentially relevant but could not be incorporated into the target SRR due to the constraint of the temporal sequence.

In the third step (Time Series Assessment), we computed and visualized the individual-level average adjusted JS time series traces of all SRRs. We colored the time series traces according to whether a time point was before or after the publication date of the SRR whose time series was being plotted. The coloring helps us to see whether the SRR's publication date marked a significant turning point in the trend. If an SRR's time series started to increase after the SRR’s publication date, then the SRR may have influenced evidence selection practices.

All computer scripts, coded using the R iGraph library version 1.5.0.1 (Csardi & Nepusz, 2006), tidygraph version 1.2.3 (Pedersen, 2023), and ggraph version 2.1.0 (Pedersen, 2022) can be found on GitHub (Fu, 2023).

Figure 3 visualizes the ExRx inclusion network. The target SRR, SRR #2, is situated in the middle of the largest weakly connected component¹¹. There are two other smaller weakly connected components. The weakly connected component at lower right (centered on SRR #23) includes 86.2% PSRs (25/29) published after the search date of SRR #2, and the weakly connected component at lower left (centered on SRR #3 and SRR #26) includes 38.4% (25/65) of PSRs published after the search date of SRR #2.

SRR #23 synthesized evidence regarding one particular type of exercise, Baduanjin, a form of medical Qigong. The most plausible reason for its isolation is that the authors of SRR #23 conducted extensive searching in Chinese-language databases (Chinese National Knowledge Infrastructure, Wanfang, and the Chinese Clinical Trial Registry) for studies on Baduanjin, and many included PSRs were written in Chinese. Therefore, language and database access may explain why SRR #23’s included PSRs were not used by any other SRRs in the inclusion network, making SRR #23 and its included PSRs an isolated cluster. Decision-makers serving a Chinese community may wish to read SRR #23 to complement SRR #2’s findings.

SRRs #3 and #26 were interested in the effect of exercise in preventing rather than treating depression. They only included PSRs reporting observational studies, and other SRRs (except #17 and #22) restricted themselves to PSRs reporting randomized controlled trials (RCTs). In this case, depending on the decision-makers’ situation, exercise’s preventive effect on depression may be considered relevant or out of scope.

In contrast to Figure 3, the target SRRs reside in their own community (Figure 4). SRRs #1, #2, #3, #5, #6, and #12 are closer together; SRRs #4, #7, #8, #9, #11, #13, and #14 are closer together; SRR #10 is distant from the others. Community formation raises concern because our target SRRs did not include evidence that had been accepted by other SRRs.

We found that the community formation aligned with SRRs’ differences in the choice of the most appropriate study design(s) to include. In Figure 5, SRRs clustered on the upper right side of the figure and within the light green shaded area are always connected to nodes representing PSRs using the RCT study design. SRRs clustered on the left side of the figure within the light orange shaded area are mainly connected to nodes representing PSRs using the cohort study design. Furthermore, SRRs on the boundary (#8, #9, and #11) are connected to both the RCT and cohort study PSR nodes, indicating that they straddle the study design division. SRR #10 (retracted) forms its own community because it studies patients with heart failure, a severe condition, whereas other SRRs, with the exception of SRR #11, study healthy or nonacutely ill people.

The study design differences arose from a struggle between inferential rigor and cost/feasibility in medical research. RCTs are often considered the highest quality evidence in health sciences for topics that permit RCTs, yet some (especially Strazzullo, D’Elia et al., 2009) argued that the RCT design was unlikely to produce definitive evidence for the salt controversy due to its inherent cost and infeasibility.

To test whether we could recover the study design differences (node attributes) through the network structure, we applied community detection algorithms implemented in the R iGraph library (Csardi & Nepusz, 2006) to the weakly connected components (Supplementary material, Table S2). Two algorithms, edge betweenness and infomap, recovered the community structure shown in Figure 5, with only a minor difference regarding the assignment of node #60. Notably, the success of the edge betweenness algorithm indicates that edge betweenness is an important metric to explore in the future¹².

Our investigation with Figure 5 revealed a significant risk of using SRR #12 as the foundation of action: Many other SRRs have moved towards using observational studies solely or in combination with RCTs to evaluate the relationship between salt intake and all-cause mortality and cerebro-cardiovascular disease. Although Cochrane reviews are known for excellence in review methodology, the prespecification of inclusion criteria can cause limitations when Cochrane reviews limit themselves to RCTs. Users interested in learning the relationship between salt intake and cardiovascular disease and stroke are strongly recommended to seek information from systematic reviews that consider non-RCT evidence.

Table 2 reports the results from a quantitative comparison between the target SRRs and their companion SRRs. We also list the interval between the target SRRs’ search date and their companion SRRs’ search date (i.e., “search date interval”): A positive search date interval means that the companion SRR’s last search date comes after the target SRR’s last search date. We sorted the companion SRRs according to their adjusted JS in ascending order.

In principle, adjusted JS should be larger than the regular JS, but there is one anomaly between ExRx SRR #2 and ExRx SRR #8 (adjusted JS = 0.133, regular JS = 0.140). The cause of these anomalies is that we use the publication date to anchor PSRs in the temporal sequence. However, a SRR can still include a PSR published after its last search date if the PSR was findable through a bibliographic database (e.g., already indexed as online first before its publication in a journal issue) or a clinical trial registry, resulting in an underestimation of adjusted JS.

By examining the first few rows of Table 2(a), we identify other ExRx SRRs that do not share any PSR with our target SRR, SRR #2, besides those we discovered from the visualization assessment step. SRR #19 is an intriguing case: It studies yoga for major depression but shared no PSR with SRR #2. It turned out that SRR #2, although its inclusion criteria did not mention yoga, or explicitly state they would exclude PSRs studying yoga, filtered out several PSRs studying yoga in practice (see “Characteristics of excluded studies” of Cooney et al. (2013)) but retained two PSRs studying laughter yoga. Due to these two PSRs on laughter yoga, SRR #2 has miniscule but nonzero adjusted JS with SRR #4, which specifically studies yoga (adjusted JS = 0.043).

Both SRR #14 and #17 are about the adolescent population and have zero adjusted JS and regular JS with SRR #2, indicating that our target SRR may have overlooked the adolescent population, which is true, because SRR #2 only studies the adult population.

SRR #9, #12, and #16 are the three ExRx companion SRRs with the top three highest adjusted JS scores with SRR #2. SRRs #2 and #9 have a high adjusted JS (0.634) and their JS fold change (1.29) indicates a considerable amount of new evidence. SRR #9 concerns the dropout rate from randomized controlled trials among people with depression, and therefore may not be of direct interest to users wanting to know whether exercise helps alleviate depression.

By contrast, the other two SRRs with high adjusted JS with our target SRR, SRRs #12 and #16, are both of particular interest. Their titles both contain the keywords “exercise” and “depression.” SRR #16 included more new evidence (JS fold change = 1.18, search date interval = 37 months) than SRR #12 (JS fold change = 1.01, search date interval = 29 months). In addition to checking SRR #16 for new evidence, users should also consider comparing SRRs #16 and #12 to understand why SRR #12 does not include more new evidence as compared to SRR #16.

By examining the first few rows of Table 2(b), we identify salt controversy SRRs that do not share any PSR with SRR #12. SRR #12 has zero adjusted JS and regular JS with SRR #7 and #14 due to the difference in study designs. SRR #12 does not have any overlap with SRR #10, the only faceting SRR in the salt controversy inclusion network, because “[d]oubts have been raised about the integrity of research from the Paterna group” (Adler et al., 2014), the group that produced SRR #10. Consequently, due to these integrity concerns, PSRs included in SRR #10 were not selected by SRR #12. In summary, we did not identify new potentially relevant evidence overlooked by SRR #12.

SRR #12 is relatively late among all salt controversy SRRs, as only two SRRs, SRRs #13 and #14, finished their search after SRR #12 (Table 2(b)). Their adjusted JS with SRR #12 are both low (0.056 for #13 and 0 for #14) and their JS fold changes are both 1 (i.e., no change). As we cannot find SRRs that (a) finished their search after the target SR; (b) have a high adjusted JS with the target SRR; and (c) have a high JS fold change with the target SRR, we did not identify potentially relevant evidence published after the search date of the target SRR in this case.

We designed the individual-level average adjusted JS to assess the stability in an SRR’s evidence selection standards: whether an SRR’s choice of evidence continued to be used by other SRRs, or whether the evidence it selected was no longer used by other SRRs. Figure 6 shows that ExRx SRR #2’s time series experienced a steady period of increase from 2014 to 2016. Thus, despite the criticism it received (Clarke, 2019), SRR #2’s choice of evidence gained acceptance. The increase also took place after its publication, which increased the significance of SRR #2: It may have influenced the evidence selection of later SRRs by increasing the exposure of the PSRs it selected. However, after 2016, SRR #2’s time series plunged, but so did the time series of many other SRRs (Figure 6). The growing dissimilarity can also be observed in Figure 3, the visualization of the ExRx inclusion network. SRRs that finished their search in 2016 or later appear either in isolated clusters (SRR #26 and SRR #23) or on the periphery of the largest weakly connected component (SRRs #18, #19, #20, #21, #22, #24, #25, #27, and #28). Because SRR #2 is situated at the center of the largest weakly connected component, the distance among the 12 SRRs indicates the dissimilarity amongst them. The 11 SRRs that finished their search in 2016 or later appear to make a concerted effort to select different evidence than our target SRRs—which are also different from each other. Our domain expert Caitlin Clarke suggested a plausible explanation for this diversification: Perhaps researchers determined that the overall topic (the relationship between physical activity and depression) was too broad and chose to further narrow their research questions and redesign their inclusion criteria for narrower research questions, resulting in disjoint sets of evidence selected.

Our target salt controversy SRR (SRR #12) shows a downward trend, along with other RCT-only SRRs (SRRs #1, #2, #3, #5, and #6) (Figure 7). Among the companion SRRs, the majority show upward trends (#4, #7, #11, #13, and #14). Two show downward trends (#8 and #9), and one is nearly flat (#10). The average adjusted JS decreased for SRRs that only include RCTs (all target SRRs) because emerging SRRs started to include observational studies. Meanwhile, the average adjusted JS increased among SRRs that include observational studies (most companion SRRs) because more peers with similar evidence selection criteria started to emerge.

We also detected salt controversy SRR #4 as a potentially influential SRR by using the time series in Figure 7: The rise of the time series took place after its publication. It was the first among all salt controversy SRRs to include observational studies. Although it is impossible to know whether SRR #4’s argument in support of including observational studies influenced others, among the subsequent 10 SRRs that completed their search after SRR #4’s publication date (November 2009), 80% (8/10) cited SRR #4, indicating their awareness of SRR #4.

Table 3 summarizes all the risks we identified through the inclusion network approach. Evaluating risks is beyond the scope of the inclusion network approach itself. In general, decision-makers must evaluate the risks before deciding whether a given review is sufficient to use as the foundation of their action.

For example, our salt controversy case study identified two risks in SRR #12: Many other SRRs included observational studies solely or in combination with RCTs, and including observational studies is a more recent trend. To investigate whether SRR #12 would be sufficient to use as the foundation of their action, decision-makers need to evaluate those two risks against the fact that SRR#12 only includes RCTs and the fact that observational studies are, in general, of lower quality than RCTs. One next step would be for decision-makers to assess the merits of observational studies for this topic.

In comparing the inclusion network approach and claim-specific citation network, we highlight several high-level similarities as well as four significant differences.

First the similarities: Both aim to assess scientific knowledge. Both can trace error propagation: by identifying citation distortion in claim-specific citation network analysis or by making evident whether any SRRs have included problematic PSRs in the inclusion network approach. Both rely on network representations and therefore are sensitive to community formation. In fact, a common risk identified by both approaches is the existence of unresolved disagreement manifested through these network structures. Trinquart et al. (2016) identified communities through their claim-specific citation network, which are “distinct and disparate lines of scholarship” that bear “little imprint of an ongoing controversy.” From the inclusion approach, we discovered community formation in the inclusion network (Figure 5), which represents disparate evidence selection standards.

Among the differences, first, in claim-specific citation network analysis, the entire network is about a single scientific claim, and the structure of the entire network is used to assess that claim. In the inclusion network approach, we assess only one systematic review (the target SRR) using the inclusion network of the target SRR and its companion SRRs. And the conclusions drawn by a systematic review are usually more nuanced and complex than a scientific claim.

Second, claim-specific citation network analysis connects the structure of the claim-specific citation network to violations of scientific objectivity: for example, biased representation of the available evidence, distorting the meaning of a claim from the source literature, or inventing a claim that was not contained in the source literature. These violations concern trust in scientific knowledge but mainly from the position of trained scientists. By contrast, we define risk inclusively as any information concerning a systematic review that can reduce a user’s willingness to use the systematic review as the foundation of their action. The judgment of risk is left to the users of the inclusion network approach because we, as designers, will never have all the knowledge pertinent to the assessment.

Third, the inclusion network approach is more analyst driven than claim-specific citation network analysis, a characteristic that we demonstrated through the two case studies above. It involves a process of sensemaking of the network, particularly in the first step, Visualization Assessment, which relies on the analyst’s attention and judgment.

Fourth, the inclusion network is better positioned to study evidence selection standards than claim-specific citation network analysis is due to the specific meanings encoded in the edges. In both networks, edges represent that a citing paper’s authors selected the cited paper. In general, authors’ decisions regarding what to cite are not explicitly documented; but in the special cases of systematic reviews and other publications that support decision-making, the selection criteria are explicitly stated in a publication’s inclusion criteria. Without being restricted to particular types of scientific publications, a claim-specific citation network provides a bird’s-eye view of a scientific belief system and is better positioned to study the overall establishment process of a scientific belief. Yet in the inclusion network approach, the edges connecting the SRR and its included PSRs in the inclusion network are a consequence of explicitly stated evidence selection standards. This enables measuring the stability of the evidence selection standards of a target SRR using the individual-level average JS time series, as we demonstrated in Section 5.3. And it enables community detection on an inclusion network to highlight disparate evidence selection standards, as shown in Figure 5. In general, the consistent correspondence between the existence of an edge and a set of documented and traceable evidence selection standards enables quantitative analysis of evidence selection standards based on the structure of the inclusion network.

Parallel to the temporal analysis by Shwed and Bearman (2010), with the inclusion network we studied the stability of the evidence selection standards of the target SRRs using the individual-level average JS time series. We must highlight that the time series analysis for evidence selection standards was made possible by the explicit evidence selection procedures and reporting standards of systematic reviews.

The inclusion network spotlights the choice of evidence, however many other factors that can influence the conclusions drawn by a systematic review were not considered, such as data extraction and statistical methods (Bolland & Grey, 2014) or the handling of heterogeneity and risk of bias assessments (Osnabrugge et al., 2015). This is a fundamental limitation of the approach. And inferring a causal relation between the evidence selected and the conclusion drawn by an SRR must proceed with care.

Choosing appropriate companion SRRs is of paramount importance. We have yet to develop a sound methodology for choosing appropriate companion SRRs. Currently, users can collect the SRRs using systematic search as Trinquart et al. (2016) did for their data set or as Clarke did for her data set (Clarke, 2019; Clarke et al., 2023) and design their search queries based on the target SRR’s research question. However, without experimenting with various search strategies to compare their outcomes, we will not know whether systematic search is the most appropriate for the inclusion network approach. Such experimentation will be a part of the future work. Furthermore, we hypothesize that the search strategy for appropriate companion SRRs should also deliberately investigate certain known risks (Cooper, 1982; Felson, 1992), which brings us to the next limitation of the current work.

We carried out the current work without a clear idea of what risks may exist, which made the process more explorative but less systematic. In the future, we plan to seek advice from decision-makers and other users of systematic reviews and other evidence synthesis literature about which risks they find most concerning. We will develop a taxonomy of risks to guide the improvement of our approach. This taxonomy will incorporate user feedback and existing taxonomies, such as the taxonomy of bias by Felson (1992). Furthermore, we will invite decision-makers and other users of systematic review literature to test our approach; we will use their feedback to improve our approach.

We solely focused on SRRs in this study. But a study of PSRs using the inclusion network construct is also a promising direction. For example, we could study the PSR utilization using the construct, following Leng’s work on research underutilization (Leng, 2018). Because many primary studies cost tremendous resources to conduct, it will be extremely valuable to know their utilization. An analysis of PSR utilization deserves a publication of its own, as the analytical methods must be carefully designed from the perspective of PSRs to fit specific analytical purposes (e.g., detecting changes in utilization). At this point, we have presented a metric called PSR-utilization rate in the Supplementary material Section 5.

Although the publication date is easy to obtain, using the publication date to rank PSRs in the temporal sequence can be imperfect because an SRR can still include a PSR published after its last search date if the PSR was findable through a bibliographic database (e.g., already indexed as online first before its publication in a journal issue) or a clinical trial registry, resulting in an underestimation of adjusted JS. In the future, we may refine how a temporal sequence is constructed by systematically researching the findability of a PSR before its publication date and identifying a more precise way to obtain the rank value for PSRs.

The most important future work is at the data infrastructure level: indexing included PSR separately, which requires community effort. Currently, we must manually extract included PSRs, assign them unique identifiers, and curate the inclusion relationship to form an inclusion network, which is neither scalable nor timely. Databases of scientific evidence and/or their synthesis, such as The Collaboration of Environmental Evidence Database of Evidence Reviews (CEEDER) and Meta-Analytic Psychotherapy Research (Metapsy) can preserve the record of inclusion relationships and make them retrievable for users. Over time, such efforts will yield large-scale inclusion networks that can be retrieved and analyzed like today’s regular citation data, as an important data source to improve the usage of scientific evidence.

We designed and tested the inclusion network approach to help decision-makers assess risks in systematic reviews they intend to use as the foundation of their action. Our approach stemmed from two lines of research, reliability issues in systematic reviews and the use of network analysis to assess scientific knowledge. It is indebted to but also significantly different from the influential claim-specific citation network approach.

We started the design process by specifying four design principles. Our design principles led us to design the data structure (the inclusion network), the network metric (adjusted Jaccard similarity), and a three-step assessment process.

We presented two case studies. Potential users should learn from our case studies how they may explore the inclusion network, move back and forth between the network and the content of the literature, and develop their own understanding about the SRRs and PSRs. For our case studies, we prepared two inclusion network data sets, each based on data initially collected by researchers interested in studying systematic reviews addressing a given topic. We used these two data sets as two case studies and assessed two SRRs: ExRx SRR #2 and salt controversy SRR #12. Risks we identified include missing potentially relevant evidence (e.g., Baduanjin and yoga in ExRx SRR #2), epistemic division in the scientific community (e.g., disagreement regarding the most appropriate study design(s) to include among salt controversy SRRs), and recent instability in evidence selection standards.

Given the potentially momentous consequence resulting from decision-makers applying our inclusion network approach to their work, we strived for an ethical, inclusive, and transparent design that respects the agency and knowledge of the decision-makers. We hope this paper provides an example for future quantitative science researchers who want to design methods for diverse users. The inclusion network approach still needs improvement, and in the near future, we hope it can increase confidence in the applicability of evidence synthesis products such as systematic reviews, or by contrast, can rationalize the commissioning of up-to-date or more expansive syntheses when the risks are deemed unacceptable.

Tzu-Kun (Esther) Hsiao coauthored our previous paper on the salt controversy and gave useful feedback on this manuscript. Thanks to George Chacko, Jana Diesner, Ly Dinh, David Hopping, David Johns, Jay Patel, Zoi Rapti, Malik Salami, and Linda Smith for comments and to Trinquart et al. for data sharing. Thanks to Maria Gillombardo for introducing CC and JS. Thanks to the anonymous reviewers of Quantitative Science Studies for their extremely helpful comments. Thanks to Randi Proescholdt for providing the Multi-Tagger classification outputs used to cross-check the study design classifications shown in Figure 5.

Mark Van Moer: Software, Visualization, Writing—review & editing. Caitlin Vitosky Clarke: Data curation, Resources, Writing—original draft, Writing—review & editing. Jodi Schneider: Conceptualization, Data curation, Funding acquisition, Methodology, Supervision, Writing—original draft, Writing—review & editing. Yuanxi Fu: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Writing—original draft, Writing—review & editing. Natalie Lischwe Mueller: Data curation. Manasi Joshi: Data curation. Tzu-Kun Hsiao: Data curation, Methodology, Visualization.

The authors have no competing interests.

This research was supported by NSF 2046454 CAREER: Using network analysis to assess confidence in research synthesis and by the Campus Research Board of the University of Illinois at Urbana-Champaign Grant RB21012. MVM’s participation was supported by the University of Illinois Research Software Collaborative Service. Multi-Tagger was funded by NIH/NLM R01LM010817.

Two data sets were used in this work: ExRx (Clarke et al., 2023) and salt controversy (Fu et al., 2023).