A long and winding road: Research impact evaluation over public policies Open Access

https://www.webofscience.com/api/gateway/wos/peer-review/10.1162/qss_a_00345

Given the ongoing social transformations across various scales and domains, such as climate change, poverty, inequalities, and technological advancements, the significance of understanding the relationship between policy and science has never been greater (Aiello, Donovan et al., 2021). The use of scientific knowledge in policy also overlaps with the growing call for evidence-based policymaking, creating a research agenda intersected with discussions on the use and impact of research on policy (Bleiklie & Michelsen, 2022; Bozeman & Youtie, 2017; Cairney & Oliver, 2020; Eden & Wagstaff, 2021; Gunn & Mintrom, 2021; Wellstead & Howlett, 2022). In this context, there is a growing concern among funding agencies, universities, and other stakeholders in the research ecosystem regarding how research can better inform, influence, and impact policies, thereby making this topic increasingly relevant (Álvarez-Bornstein & Bordons, 2021).

The discussion about the relationship between research and policy also implies discussions about methods to assess the impacts of research (Álvarez-Bornstein & Bordons, 2021). Among the various methods to identify and measure the impact of research on public policies, bibliometric and altimetric methods can offer interesting insights (Herzog, Hook, & Konkiel, 2020; Pinheiro, Vignola-Gagné, & Campbell, 2021). However, despite the increasing use of such approaches, there is still a large field of possibilities for empirical and methodological discoveries to be explored, as pointed out by several authors, such as Hicks and Isett (2020) and Romero-Frías, Torres-Salinas, and Arroyo-Machado (2023).

First, there are gaps in empirical evidence on the use of research in public policies in various national contexts, particularly in the Global South, due to geographic or language biases. Second, the methodological possibilities arising from the identification of research in secondary sources used in the processes of policy formulation, implementation, and evaluation are quite broad. Today, it is almost impossible to track the exact influence that an article makes on a policy using secondary data. There are challenges both in the identification of the use of research in policy documents and in extracting and organizing this information. Tools such as Altimetric.com and Overton, for example, offer interesting possibilities, although limited, as we are going to argue (Pinheiro et al., 2021).

We aimed to examine the influence of FAPESP, a leading research funding agency in Brazil, on policymaking through the research it supports. We sought to utilize the Overton tool to identify how FAPESP-funded research impacts policies, framing two research questions: What kind of impact of FAPESP in shaping policymaking can be measured using the Overton database? What limitations and challenges lie ahead in refining the evaluation of the impact of research within policy documents?

Thus, the article presents results from a study that took into account data obtained from articles—through their Digital Object Identifiers (DOIs)—stemming from programs and funding support from FAPESP to produce novel insights about the impact of knowledge production on policies.

The article proceeds with a brief conceptual discussion on evaluating the impact of research on policies, focusing on bibliometric approaches; it then details the methodological procedures used in the research, followed by describing and analyzing the results. In conclusion, beyond highlighting the obtained results, the potentials and limitations of the approach are emphasized, aiming to raise new research questions and discuss the practical implications of the findings.

The relationship between research and policies is complex and nonlinear. Researchers, research products, and policymakers can engage for different reasons, forms, and contexts and at different stages of the policy cycle (agenda, design, decision, implementation, evaluation, redesign, etc.). In policies, research becomes just another source of knowledge among others, and it blends with multiple values, ideologies, institutions, laws, power relations, public opinion, conjunctures, actors, and political groups, and so on (Louder, Wyborn et al., 2021; Williamson, Barker et al., 2019). Here, the same elements that make it difficult to evaluate the social impact of research apply to policies: temporality (short, medium, and long term), different types of research in different areas of knowledge, and the challenge of identifying causal relationships, among others (Edwards & Meagher, 2020; Lauronen, 2020; McCowan, 2018; Ozanne, Davis et al., 2017). Researching the relationship between research and policies constitutes a significant methodological and analytical challenge (Boaz, Fitzpatrick, & Shaw, 2009; Edler, Karaulova, & Barker, 2022; Sordè Martí, Flecha et al., 2020).

The literature has pointed out a wide range of ways to identify, evaluate, and measure how scientific knowledge flows from its production spaces to engage and influence policy (Andrews, 2007; Bornmann, 2013; Jones, McBeth, & Shanahan, 2014; Lacey, Howden et al., 2018; Ozanne et al., 2017; Smit & Hessels, 2021; Viana-Lora & Nel-lo-Andreu, 2021). Overall, studies have adopted a variety of theoretical and methodological perspectives from two nonmutually exclusive paths (Newson, King et al., 2018; Newson, Rychetnik et al., 2021): forward tracking and backward tracking. The first path starts from the research products of researchers, research groups, universities, or agencies to identify, evaluate, and analyze how research has been incorporated into policies. In the second path, one starts from the policies themselves, the institutions and actors that produce them, or policy documents to identify, analyze, and evaluate the use and impact of the knowledge used. In this sense, the starting point is important for assessing the impacts of research on public policies, even if the paths are not mutually exclusive.

Posner and Cvitanovic (2019) argued that there are five main methods for collecting data for evaluating boundary-spanning activities focusing on the impact of research on policies: interviews, which tend to deepen the perspectives of the involved actors; questionnaires, which usually allow collecting a larger volume of data with less depth; network analysis, which allows measuring the strength of interaction among involved stakeholders; content analysis, where documentary sources are analyzed to find evidence of impact; and rapid assessments, which prioritize a more panoramic and superficial collection and analysis of various data, focusing on producing quick evaluations. The authors emphasize that the types of data and collection procedures depend on the objectives and the time/resources available for evaluation.

In this scope, it is worth noting the use of logical and multidimensional evaluation models, such as the Payback Model and the Research Impact Framework, among other models (Donovan & Hanney, 2011; Edwards & Meagher, 2020; Milat, Bauman, & Redman, 2015). The literature on research impact assessment in policies has shown the existence of a diversity of approaches. First, there are models applied by university governance bodies and research centers (Alla, Hall et al., 2017). The United Kingdom’s Research Excellence Framework (REF), for example, is one of the most well-known cases where assessment panels are based on narratives of social impact (including in policies) produced by the researchers themselves (Greenhalgh, Raftery et al., 2016; Parker & van Teijlingen, 2012; Torrance, 2020), which are subsequently verified. The REF, as shown by Yaqub, Malkov, and Siepel (2023) establishes a sort of ex-ante to ex-post tracking that allows the verification of research impacts on policies.

Other cases with a significant presence in the literature and evaluation practice can be observed in the health sector, recognized as a field of examples (Greenhalgh & Fahy, 2015; McElfish, Purvis et al., 2018; Vilkins & Grant, 2017). It is more and more common to assess the impact of research on health-tracking clinical protocols through their sequential phases.

Louder et al. (2021) present a synthesis of proposals, discussions, and frameworks for assessing the impact of research on policies in the environmental science area. The results point to different ways to define and measure the impact of research on policies. Some approaches tend to focus on stakeholder interaction and the changes produced by research (especially in terms of knowledge gain and changes in stakeholder practices) and vice versa (how interaction produced a change in the researcher and the research process), considering a processual dimension. Other assessments tend to focus on identifying the types of research impact on policy (e.g., instrumental, conceptual, capacity building, strategic use, formation of new networks, etc.) and also the levels of impact (individual, organizational, policy, and practice).

Within these paths, the use of qualitative, quantitative, and mixed approaches is noted, following data collection procedures that mainly involve documentary research, conducting interviews, evaluation panels, questionnaire administration, and the use of bibliometrics and altmetrics (Milat et al., 2015). Depending on the methodological focus, data collection may focus on researchers and their institutions or on the actors involved in the policy design, decision-making, implementation, and evaluation processes.

Other assessments may understand the impact of the production of new products within the scope of policies (protocols, reports, regulations, etc.). However, the literature acknowledges that this approach misses central elements of interaction, and many policy documents do not directly present or cite research utilization (Isett & Hicks, 2020). These approaches, along with issues such as temporality and the scope of evaluation, are key factors in defining impact metrics and indicators.

In their study, Ozanne et al. (2017) delved into the intricate process of knowledge transfer between researchers and relevant stakeholders. The authors advocate for broadening the conventional strategies and metrics used to gauge research impact, such as bibliometric approaches, to encompass process evaluations tailored to individual stakeholders throughout the entire trajectory of impact—from knowledge creation and awareness to utilization and societal repercussions (Ozanne et al., 2017). Similarly, Viana-Lora and Nel-lo-Andreu (2021) examined the principal methods and tools employed to evaluate the social impact of research, including case studies, alternative metrics, productive interactions, interviews, and evaluation reports. The authors highlight the usefulness of alternative metrics and altmetrics for estimating and measuring the use, circulation, and popularity of research (such as when it is mentioned in news outlets). While they are useful and can complement other methods, the authors point out that this approach also presents challenges in measuring and assessing impact, as these metrics do not show social changes and impacts. Their investigation indicates that assessments primarily reliant on citation counts fail to capture the dynamics of researcher–citizen interaction.

Nevertheless, the utilization of bibliometric and altimetric analyses to assess the nexus between research and policies has experienced significant development (Murat, Noyons, & Costas, 2023; Szomszor & Adie, 2022; Tahamtan & Bornmann, 2020). One of the research and evaluation paths is important to highlight concerns about the use of bibliometric and altimetric analyses to assess the relationship between research and policies (Tahamtan & Bornmann, 2020). This approach usually involves keyword searches or text mining techniques to identify references and mentions of research, universities, or researchers in policy documents. The approach may involve backward or forward tracking, using the article or so-called policy documents as the main units of analysis (Bogenschneider, Day, & Parrott, 2019; Vilkins & Grant, 2017).

Some tools and databases have been developed for this purpose, such as Altmetric.com and Overton (Bornmann, Haunschild et al., 2022; Maleki & Holmberg, 2022; Pinheiro et al., 2021; Szomszor & Adie, 2022; Yin, Gao et al., 2021). They start from indexed policy documents to identify, through text mining, mentions of research, normally using a digital identifier. Recent research shows that such databases index documents to a large extent differently from each other (Murat et al., 2023). Murat et al. (2023) compared policy document sources indexed by Altmetric.com and Overton, and their findings indicate that Overton covered more policy organizations than Altmetric.com, albeit with a relatively small overlap in sources. The authors consider that this small overlap may indicate that both data aggregators may be employing slightly different operationalizations of their definition of “policy documents.”

Despite its relatively recent emergence, the Overton database has quickly become a potential tool for understanding public policy through its comprehensive indexing of policy documents. Literature using Overton includes quantifying cross-domain knowledge diffusion from science to policy (Pinheiro et al., 2021; Ren & Yang, 2023), analyzing which research contributed most to climate change policy (Bornmann et al., 2022), and explaining the coevolution of policy and science during the Covid-19 pandemic (Yin et al., 2021). Other contributions also include analyzing the impact of local knowledge in public policy (Jonker & Vanlee, 2024; Maleki & Holmberg, 2022), tracking how tobacco-related social media research has been cited in government policy documents (Beard, Donaldson et al., 2024), measuring the influence of open science on public policy (De Filippo & Sastrón-Toledo, 2023), exploring the use of predatory journals articles in public policy (Brandts-Longtin, Lalu et al., 2022) and analyzing how the lack of a consensus definition in the literature affects policy documents citations (Dibbern, dos Santos Alves et al., 2024).

Considering research endeavors aimed at exploring the potential and limitations of that platform, we highlight the studies from Szomszor and Adie (2022) and Maleki and Holmberg (2022). Szomszor and Adie’s (2022) investigation reveals a prevalence of the Overton tool regarding certain research topics such as health, economics, social care, and the environment, unlike conventional bibliometric databases that usually focus more on life and natural sciences. Regarding the database coverage, Maleki and Holmberg (2022) compare the coverage of and overlap between Altmetic.com and Overton policy citations. The study compared the policy coverage of 18,996 Scopus publications authored by researchers affiliated with 18 Finnish universities and institutes. The results indicated that combining results from both sources can provide the best reflection of citations in policy documents; however, to get a national account of policy implementations, Overton outperforms Altmetric.com and provides more promising citation coverage.

FAPESP is a funding agency linked to the Department of Science, Technology, and Innovation of the State of São Paulo, Brazil. In Brazil, there are public funding agencies linked to the Ministry of Science and Technology, with nationally scoped programs (covering all 26 states in Brazil). The two main Brazilian agencies in this field are The National Council for Scientific and Technological Development (CNPq) and the Coordination for the Improvement of Higher Education Personnel (CAPES). In addition to these, there are funding agencies at the states level (State Foundations in all 27 Brazilian states) that make up the federation of funding agencies called State Foundations for Research Support. These foundations focus on fostering research and innovation in their respective states and are linked to the corresponding governments. In terms of budget, number of research projects, applicants, and scientific production, FAPESP may represent around 50%, with the other half unevenly distributed to the other state funding agencies. On the one hand, its resources come from budget allocations from the government of the State of São Paulo, which has the highest Gross Domestic Product compared to all states, and the region is considered the most economically active and industrialized in the country. For example, in 2021, while CNPq allocated 1.2 billion reais (CNPq, 2021), FAPESP alone allocated 1.85 billion reais exclusively for the state of São Paulo (Agência FAPESP, 2021).

Our data are based on published articles stemming from projects and scholarships funded by FAPESP. The data were obtained from the agency’s public data source called Virtual Library.

We have retrieved 99,637 DOI records from 52,425 funded projects and/or scholarships for master’s, doctoral, and postdoctoral studies, spanning from January 1992 to February 2023. The articles are distributed as follows, according to FAPESP’s classification of knowledge area: 0.1% in Linguistics and Arts; 0.5% in Applied Social Sciences; 1% Interdisciplinary; 1.7% in Humanities; 10.5% in Engineering; 11.8% in Agricultural Sciences; 19.7% in Health Sciences; 26.1% in Exact and Earth Sciences; and 28.6% in Biological Sciences.

The predominance of articles in certain areas of knowledge was noted beforehand, which is important for the results obtained. Most of the studies that have utilized Overton and Altmetric.com conducted searches in Overton starting from traditional databases, such as Scopus and Web of Science (Pinheiro et al., 2021). We gathered data directly from the funding agency database under study, which provided us with a high level of reliability regarding the link between the funding source (FAPESP) and the scholarly output².

After searching for DOIs in Overton, the data were extracted via an application programming interface (API) and integrated into an Excel spreadsheet. After data cleaning procedures (checking and correcting any potential duplications or inaccuracies), all downloaded data were integrated into a master spreadsheet. This master spreadsheet combined the input data obtained from the FAPESP Virtual Library with information related to the articles (DOIs authors’ affiliation, articles’ titles, abstracts, authorship, keywords), along with data obtained from Overton. This procedure allowed the construction of an integrated database that enables linking between the funding input profile and the results obtained in Overton. A set of 2,993 publications mentioned in the Overton database was found.

To describe and analyze the results, we chose to start by exploring the profile of the identified articles mentioned in policy documents. We found it important to consider the profile of the mentioned DOIs as it may reveal significant aspects of their use in policies. For the quantitative analysis, procedures related to bibliometrics were adopted (Glänzel & Schoepflin, 1994; Tague-Sutcliffe, 1992), using the Python language as a technological resource. In this bibliometric analysis of publications supported by FAPESP, we measured the following indicator. First, productivity indicators were considered (temporal distribution of publications, publication channels, types, and areas of funding). Bradford’s Law, used in measuring the behavior of scientific literature (Bradford, 1934), was applied to identify the main publication channels. Second, in addition to the impact indicators, citation indicators from the SciVal platform were also extracted. SciVal is a platform from Scopus (Elsevier) that offers various metrics on research performance, collaboration, trends, and comparisons (whether for a set of articles or research organizations). All the data generated were exported in .xlsx format and graphically represented using Microsoft Excel. In a second step, the profile of policy documents mentioning the set of DOIs was described and analyzed, also utilizing bibliometric techniques (publications cited in policy documents, based on variables such as institution, region, country, and citing organization). All the data generated were exported in .xlsx format and graphically represented using Microsoft Excel. Here, insights from Overton show us the sources producing the documents, taking into account the type and geographic distribution.

After the bibliometric analysis, thematic segmentation of the abstracts of publications supported by FAPESP and cited by different types of organizations—government, intergovernmental organizations (IGOs), think tanks, and others—was conducted (see Section 5). This was done using an unsupervised machine-learning technique with the Orange software (https://orangedatamining.com/). The process was divided into three stages: text preprocessing, document embedding, and clustering via the K-means algorithm. In the document embedding step, all abstracts of the articles were mapped by a pretrained model (multilingual SBERT) that identifies semantics and textual contexts, transforming them into numerical vectors (multidimensional vectors), allowing the machine learning model to process and analyze the texts, now represented as nearby vectors. With the transformation of documents into vectors, the second step employed the application of the K-means algorithm. This algorithm segments the abstracts into groups based on vectorial characteristics, seeking a configuration that minimizes internal variance and maximizes distinction between the groups. Although K-means generated clusters with similar characteristics, we observed that these clusters did not perfectly align with the pre-existing categories of the documents. Therefore, the content analysis of the abstracts was conducted using predefined categories (think tanks, IGOs, others, and government), which allowed for a better understanding of the thematic and semantic approaches of the FAPESP-supported articles.

We identified 2,993 scientific articles cited by 2,988 policy documents. In fact, there was an almost identical ratio of one article cited in one policy document. Also, the results of the identified articles represent around 3% of the initial input (2,993 out of 99,637 DOIs). Pinheiro et al. (2021) found a proportion of 5.8% of Scopus papers (2008–2016) cited in policy documents. Even though our initial sample came from a different database, and considering that it is not our intention to determine whether the results show a high or low degree of citation in policy documents, it is worth considering some limitations related to the sample that may explain the approximately 3% citation rate. In fact, there are papers dating back to 1992 in the initial input, although they are a minority, and there is an initial bias in the areas of knowledge in the input (a smaller proportion of papers in areas such as Linguistics and Arts; Applied Social Sciences; Interdisciplinary; and Humanities). It is also important to note that the input of 99,637 papers was drawn from the Web of Science through work by the FAPESP Virtual Library, and the agency’s team (which provided the inputs) estimates that there is an underrepresentation of articles prior to 2009. The results are shown in the following subsections.

The temporal analysis covered 18 years, from 2005 to 2023, and revealed an increase in the volume of publications referenced in policy documents until 2018 (Table 1). Notably, the share of cited articles rose steadily from 2005, reaching its highest in 2011 at 5.66%. This is somewhat to be expected, considering that it takes some time for articles to be cited by other documents, be they other articles or policy documents.

The 2,993 articles identified were published in 1,346 scientific journals. The distribution of the zones of these journals, based on Bradford’s Law, indicates that zone 1 (core) is made up of 96 journals (7%), zone 2 (intermediate) of 347 (26%), and zone 3 (dispersion) of 903 journals (67%). Hence, many journals were used as publication channels, but only a few (core) were the most significant in terms of publishing the results of research funded by FAPESP. Of these, 20 journals with more than 11 articles published stand out, constituting 15% of the total (Figure 1). The decline in 2022 is due to two factors: the impacts of the Covid pandemic and issues related to the extraction of data from FAPESP in WoS, which occurred at the beginning of 2023 (therefore, the data from 2022 and 2023 should be taken with this caution).

The main journals for publication notably include high impact factor (IF) journals such as Science (IF: 56.9) and Nature (IF: 64.8), both founded in the 1800s and published weekly with major contributions in various scientific fields. Other journals frequently cited in policy documents, such as Environmental Science and Pollution Research (IF: 5.8), Chemosphere (IF: 8.8), and Journal of Cleaner Production (IF: 11.1), point to a research community actively engaged in pressing environmental and sustainability issues, areas of critical policy relevance.

The resources for the development of research, subsequently published in scientific articles, come from 55 types of FAPESP funding, with nine types accounting for 81 percent of all occurrences (Table 2).

The analysis of funding sources further reveals a heavy reliance on “Regular Research Grants” and various scholarship programs, which collectively account for over half of the funding instances. Additionally, the scholarship programs within the country for postdoctoral fellows, PhD, and direct PhD candidates collectively made up 28.5% of the total funding. Moreover, funding aimed at Thematic Projects³, including programs such as Global Climate Changes, comprised 13% of the total, which, as can be seen later in this article, is a frequent theme of policy documents linked to the set of articles of our sample.

To complement the data, we searched for metrics on the scientific impact of the 2,993 articles cited in policy documents. To do that, we extract indicators from the SciVal platform, comparing the initial input of DOIs with those articles mentioned in policy documents (2,993 out of 99,637). Table 3 presents the results.

As can be seen, articles mentioned in policy documents have higher metrics for the indicators in Table 3 than the total set of publications stemming from FAPESP’s funded research between 2005 and 2023. In particular, the sample of mentioned articles has 2.5 times the FWCI; circa three times the number of citations per publication. Also, the mentioned articles have higher percentages of international collaboration (51.3% against 40.3% of the total input) and higher open access status (59.1% against 53.8% of the total input). Complementing these metrics, it was also possible to see that the set of 2,400 DOIs recognized in SciVal (out of 2,993 cited in policy documents) has higher international collaboration (based on author affiliations). The data show that 36% of the authors are from Brazil; affiliations in the United States represent 19%; France-based authors account for 12%; and the United Kingdom accounts for 9% of the authors.

The total imputed DOIs show a different pattern regarding collaborations and institutions: Brazil-based authors account for 64%, followed by France (11%) and the United States (9%). Even though the top institutions in both sets show similarities when it comes to Brazilian institutions, with some Brazilian universities and research institutes/companies at the top (such as the University of São Paulo, São Paulo State University Júlio de Mesquita Filho, University of Campinas, and the Brazilian Agricultural Research Corporation), universities and research institutions from other countries appear more prominently at the top of collaborations in the set of 2,400 DOIs. For example, prominent collaborators include CNRS and the Institut de Recherche pour le Développement from France, Harvard University from the United States, and the University of Oxford from the United Kingdom.

These data underscore our analysis that the sample of articles cited in policy documents possess high scientific impact within their respective fields, as well as indicating a more globalized profile of the research output. In other terms, we found that the mentioned articles have a relatively high academic impact in their fields and are published in internationally recognized journals with a more internationalized profile of collaboration with authors from recognized institutions.

An analysis of the most prevalent topics in the total input and the identified sample reveals a general similarity in themes (environment and health predominate), as Table 4 shows.

What differs between the two is that topics related to the environment and food security appear with greater prominence in the identified sample of policy documents, as well as specific differences with more applied issues. A general analysis also reveals that both sets show proximity (whether in topics related to the environment or health) to issues relevant to Brazil and Latin America, as well as applied studies.

Overall, the data raise questions that may partially explain this international flow of the use of articles in policy documents. On one hand, these are articles with high scientific impact in their fields, which may suggest a factor of scientific credibility (Conway, 2021) also associated with their use in policies. On the other hand, the characteristics of the organizations (see below) that mention FAPESP articles involve actively seeking scientific expertise for the later of policy documents, as these organizations, to a certain extent, aim to build knowledge in order to bridge the gap between research and policy (Isett & Hicks, 2020).

In total, 2,988 policy documents were identified, citing the 2,993 articles from the pool of 99,637 DOIs inputted into Overton. Most of the policy documents range from 2016 to 2022 (90% of the policy documents identified). The analysis reveals that 350 institutions cited articles financed by FAPESP. Among these, 16 institutions stand out, accounting for 51% of the total cited articles (Table 5).

Two institutions, Guidelines in PubMed Central (n = 235 citations) and the Food and Agriculture Organization of the United Nations—FAO (n = 200 citations), were prominent in the number of citations, representing 14.5% of the total. Other institutions, such as the European Food Safety Authority (EFSA) (n = 186), the Publications Office of the European Union⁴ (n = 139), and the Government of Brazil (n = 124) accounted for 15% of the total citing institutions of articles financed by FAPESP and stood out among all other institutions. These figures gain strength when considering the types of organizations citing this scientific production supported by FAPESP. Of the total funded articles, 56% were cited by governmental entities (governments, ministries, departments), suggesting that the results of this research are significant for the establishment of public policies and scientific development. Additionally, these works gained importance beyond national borders, as 23% of the articles were cited by IGOs. Think tanks (n = 344 citations) accounted for 11.5%, and Other Entities (n = 288) accounted for 9.5% of the total. These organizations may draw on research supported by FAPESP to underpin reports on political, economic, and social analysis.

Overall, the data demonstrate that articles are cited by international organizations and governments from various countries, indicating an international flow of research use in policymaking. These organizations address topics such as public health policies, food security, agricultural and food production, sustainable development, and international policy cooperation.

For instance, Guidelines in PubMed Central is responsible for publishing documents such as Clinical Guidelines. The impact of research in clinical guidance documents is significant for research funders because these guidelines combine critical and influential research to form evidence-based instructions; they represent a valuable measure of a research’s impact. When research supported by funders gets referenced in national or international clinical guidelines, it strongly indicates the research’s potential to shape policy and practice (Kryl, Allen et al., 2012). The issue of food safety also stands out, particularly within the scope of the European Union and the Food and Agriculture Organization (FAO) of the United Nations. Once again, the international impact flow of research funded by FAPESP is noted. The FAO is a specialized agency of the United Nations whose objective is to help countries combat hunger, with a transversal role in issues related to sustainability and health. Studies have shown that the FAO indeed directly influences food security policies in various countries around the world (Coll, Libralato et al., 2013; Dercon, Mabit et al., 2012).

It is not our intention to delve into each case but rather to highlight the use of research funded by Brazil (specifically, by FAPESP) that goes to important multilateral organizations. The profile of the articles helps us explain this phenomenon, as they are English-language publications with relatively high impact in their fields. On the other hand, IGOs are also sources of academic expertise that often compile and rely on scientific evidence for the preparation of their reports, recommendations, and guidelines for best practices/policies (Littoz-Monnet, 2017).

These institutions are geographically situated in 11 world regions, in addition to IGOs (20%), with Europe (41.5%), North America (22%), South America (6%), Oceania (3.5%), Southeast Asia, Western Asia, Africa, South Asia, East Asia, Central America, and North Africa (less than 1% each), distributed across 60 countries (Figure 2).

Notably, the United States is the country with the highest number of policies that cite articles funded by FAPESP, accounting for 28% (n = 543) of the total. France, the United Kingdom, Brazil, Germany, Canada, and Australia also stand out. Together with the United States, they represent almost 80% of the total citations. The overall number of policy documents from Brazil is relatively modest and remains low compared to those from foreign entities and organizations. In particular, it is observed that articles funded by FAPESP were mentioned in policy documents from North American, European, Asian, African, and Oceania countries and contributed, to a lesser extent, to the establishment of national and Latin American policies. These results should be considered taking into account Overton’s coverage inequality (which we will specify below), as it indexes more policy documents from countries like the United States and the United Kingdom.

Finally, regarding the thematic profile of the policy documents, we utilized Overton’s classification method, which identifies the main themes of a document by analyzing its phrases and entities. Overton compares these with data derived from Wikipedia pages, selecting topics based on language overlap. This method results in a broad set of potential topics for each document, from general to very specific themes. In our analysis, we focused on these general themes to determine if the articles funded by FAPESP are cited in documents with thematic similarities, comparing these with the general distribution of topics across the entire Overton database (Table 6).

In our sample of documents citing FAPESP-funded research, health-related topics dominate, with “Health” recording 925 documents, “Medical Specialties” at 701, and “Medicine” at 658. This focus extends to “Clinical Medicine” and “Health Sciences.” Comparatively, “Health” appears 1,858,553 times in the entire Overton database, way below other topics such as “Government,” “Politics,” “Human Activities,” and “Law.” This stark contrast suggests that while health is a prevailing theme in our specific sample, it constitutes a smaller proportion of the total distribution in Overton’s broader data set.

Environmental themes are also prevalent in the FAPESP-funded sample, including “Natural Environment,” “Agriculture,” “Biodiversity,” “Water,” and “Climate Change.” Again, when compared to the broader Overton database, these topics appear significantly less frequently in relation to dominant themes like “Government” or “Politics.” This demonstrates that while the overall distribution of topics in Overton reflects a wide range of policy interests, the research funded by FAPESP has a specialized impact on shaping discussions in particular thematic areas, such as health and environmental policies.

Overall, our analysis demonstrates an international flow of research funded within the Brazilian context, reaching various countries around the world and IGOs, notably in the Global North. Thus, our research also aligns with the literature by demonstrating that a large portion of these policy documents originate from organizations classified as IGOs⁵. This opens an interesting discussion that warrants further exploration, specifically regarding the potential inequalities involved in this flow and the appropriation of research by policy organizations. This body of policy documents generally resembles gray literature and other types of documents that support policies (Isett & Hicks, 2020; Pinheiro et al., 2021). The results suggest that the international flow of research use could also be explained by the type of organization that cites FAPESP-funded articles, because IGOs (and think tanks) seeks recognized/credible scientific expertise in order to produce their recommendations and to bridge the gap between research and policy (Isett & Hicks, 2020; Littoz-Monnet, 2017).

Using Overton as a tool to identify the use of research in policies fits into a broader debate about bibliometric and altmetric approaches to assess the impact of research on policies, especially considering policy documents as intermediaries (Hicks & Isett, 2020). Supporting the literature, our research managed to identify the sources of such documents and understand the profile of the mentioned research. However, this method also presents challenges in dealing with the diversity and heterogeneity of policy documents indexed according to the tool’s definitions and operationalizations (Murat et al., 2023). In this sense, this method of identifying and evaluating the use of research in policies presents challenges, such as establishing causal relationships and generalizations from bibliometric data, as data on research mentions do not provide insight into how they were utilized (Yu, Murat et al., 2023). Besides, there is also a challenge related to the fact that these databases are not openly accessible and have limited coverage concerning policy documents in various countries, including Brazil. For instance, as of May 24, 2024, Overton had indexed 11,637,794 policy documents. With the United States leading at 37.2%, followed by IGOs at 18.8%, and other significant contributions from the United Kingdom (6.3%), Japan (5.7%), and Spain (5.0%), Brazil’s representation is notably smaller, accounting for only 1.0% of indexed documents.

However, the method proved to be highly relevant in generating novel and comprehensive empirical data on research funded by a local agency in Brazil. In this sense, our results corroborate those of Pinheiro et al. (2021), as such methods and tools are a valuable enhancement of the quantitative tools used in altmetrics and other approaches for tracking societal research outcomes. Thus, although these documents may not completely reflect the extent of societal or policy changes prompted by the findings in peer-reviewed publications, they probably signal a gradual rise in the likelihood of policy impact beyond the referenced peer-reviewed article.

Nevertheless, our results also address a significant gap in similar studies by exploring the context of the Global South. The results demonstrate the use of FAPESP-funded research in topics of local, national, and global social relevance (particularly in health, environment, and food security). This is noteworthy because it aligns with the agency’s funding efforts (translated in the funding of projects such as BIOTA, BIOEN, and FAPESP Climate Change Program, which are environmental and health oriented), which are based on academic excellence and social relevance, and in recent years have been specifically directed towards the themes identified in our study (Chapman, 2022). This is also connected to the fact that the use of this research in policies stems from various funding efforts and programs that ultimately integrate (as shown by the data in Table 2), such as general grant programs, thematic programs, young researcher grants, and PhD scholarships.

The content analysis was conducted on 2,017 identified articles (976 out of 2,993 articles did not have abstracts). This stage allowed for understanding the thematic approaches of articles supported by FAPESP and cited by policy documents according to types of organizations: Government (1,212 abstracts), IGO (326 abstracts), Think Tank (267 abstracts), and Other (176 abstracts). The content analysis was conducted on 2,017 identified articles (976 out of 2,993 articles did not have abstracts). This stage allowed for understanding the thematic approaches of articles supported by FAPESP and cited by policy documents according to types of organizations: Government (1,212 abstracts), IGO (362 abstracts), Think Tank (176 abstracts), and Other (267 abstracts) (Figure 3).

The analysis of government documents demonstrates the highlighted presence of articles with the theme of health. The themes also reveal that the articles cited by government sources also possess, to some extent, a nature of applied research and are oriented towards risk management and data-driven decision-making in health policy (revealed by terms such as “group,” “treatment,” “patient,” and “risk”). On the other hand, the articles used by IGOs tend to align more with themes related to human food safety and the conservation of animal species. Health-related terms also appear in articles cited by IGOs, especially those related to disease treatment and control in tropical environments. Regarding think tanks, the articles mentioned by this type of organization demonstrated a focus on biodiversity and conservation studies, especially related to contexts of environmental changes and effects on flora and fauna. Additionally, think tanks also utilize articles related to health and food. Last, the articles cited by the source Other concentrate on the field of health, with a substantial focus on clinical studies, with contexts of treatments and therapies for specific conditions within the healthcare scenario. These terms, therefore, establish an understanding of studies primarily related to the field of health through evidence-based clinical practices.

Overall, the content analysis demonstrates an alignment between the bibliometric profile of the cited articles and the areas of activity of the political organizations citing them (government, IGOs, think tanks, and other), with a predominance of themes related to health, food, and environmental preservation. What is most evident is that sources labeled as “Other” (which largely represent PubMed documents) and the government tend to use more research on health-related topics. At the same time, IGOs and think tanks utilize more articles related to food security and the environment. However, the content analysis also reveals that, despite the majority of political organizations citing the articles as being outside Brazil, a significant portion of the articles focus on the Brazilian context or topics relevant to the national context. This demonstrates that this internationalized flow of article usage in policy documents does not necessarily mean that the focus of research funded by a local agency in Brazil is outward. Still, it is more likely that the studies have an applied nature that can be utilized in other contexts. It is important to highlight that this profile is also reflected in the input of DOIs funded by FAPESP, as the agency allocates more resources to projects in the areas of Health Sciences, Exact and Earth Sciences, and Biological Sciences. This funding allocation likely contributes to the prevalence of articles in these fields within the citation patterns observed in policy documents. Other organizations that expressly mobilize scientific research are the think tanks. The results demonstrate a strong correlation between think tanks and the environmental sector, according to the content analysis.

This study offers an analysis of the influence of FAPESP-funded research on policymaking. We explored the Overton database and used DOI data from articles that stemmed from research grants funded by FAPESP. We then integrated the results obtained from Overton with bibliometric data of articles mentioned in policy documents to understand the profile of the cited research. In total, we identified 2,993 articles cited in 2,988 policy documents. The research indicates that FAPESP-funded research contributes to national and international policy discussions, particularly in fields such as health, environmental sustainability, and food security. The widespread citation of this research across diverse geographical regions highlights its global impact and the relevance of Brazilian science in addressing universal challenges (at least concerning these fields).

In addition to the thematic analysis of the cited articles, we find it relevant that the most cited articles generally have a greater scientific impact and show more international collaboration than the initial sample of almost 100,000 articles, which may partially explain the international influence of those articles. Another important factor relates to the organizations using knowledge produced by FAPESP-funded research, which are active in seeking scientific expertise in the areas of health, environment, and food security.

The methodological approach, centered on the integration of DOI data from FAPESP-funded articles with bibliometric analysis, reveals both the geographical spread and thematic focus of policy documents citing this research. However, it acknowledges limitations inherent in the use of the Overton database, such as potential geographic and linguistic biases. Despite these challenges, the study demonstrates the broad scope and depth of policy influence exerted by research funded by FAPESP. The depth is evidenced by its integration into a variety of policy stages, including agenda-setting, policy formulation, and implementation across diverse sectors such as health, environment, and food security. Research supported by FAPESP has contributed to shaping national and international regulatory frameworks, clinical guidelines, and environmental conservation efforts, indicating not just a citation but an impact on the formulation and execution of policy decisions. This underscores the importance of integrating high-quality scientific research into policymaking, emphasizing the global interconnectedness of scientific and policy domains.

The limitations of this article are twofold. The first concerns the possible coverage deficit of the document source used for Brazilian policy documents—as well as for other countries outside the Global North. This fact makes it difficult to obtain a more comprehensive answer to the first research question addressed here, about the impact of the articles in the sample on policies.

The second limitation relates to identifying the form, direction, and degree of impact that the articles mentioned in policy documents have on the design of these policies. Here lies a critical methodological challenge for the future of research on evaluating the impacts of science on public policies: qualifying the type of impact that the articles have on the design of policy documents. Considering these limitations and the articles cited in policy documents, we were able to explore the extent to which FAPESP-funded research is utilized in policy rather than providing a comprehensive measure of the overall policy impact of articles from FAPESP-funded research. The expansion of Overton’s coverage (and other similar tools), along with other methods to measure and assess the social impact of research, should be employed to draw more conclusions. Additionally, the article offers a useful approach (a methodology) that can be applied to other funding agencies.

Future research may benefit from our results and methodology, as it lays the groundwork for further exploring the nuanced dynamics of science’s influence on policies. It invites an expanded investigation into the mechanisms through which scientific outputs are integrated into policymaking processes. This analysis can also be adapted to assess the impact of other funding agencies on policymaking, providing insights into their influence on global and local policy landscapes as well. Moreover, it is important to consider the recent creation of Overton, which is constantly expanding. Thus, we suggest future research to keep track of changes in the platform to see if participation in the overall number of policy documents changes, including the role played by funding agencies in this scenario.

Evandro Coggo Cristofoletti: Conceptualization, Investigation, Methodology, Validation, Writing—original draft, Writing—review & editing. Sergio Salles-Filho: Conceptualization, Validation, Writing—review & editing. Yohanna Juk: Conceptualization, Investigation, Methodology, Validation, Writing—original draft, Writing—review & editing. Bernardo Pereira Cabral: Conceptualization, Investigation, Methodology, Validation, Writing—original draft, Writing—review & editing. Karen E. F. Pinto: Conceptualization, Investigation, Methodology, Writing—original draft, Writing—review & editing. Sandra Holanda: Conceptualization, Validation, Writing—review & editing. Carlos Graziani: Data curation, Methodology. César Antonio Pereira: Methodology, Validation, Writing—review & editing.

The authors have no competing interests.

This work was supported by the Sao Paulo Research Foundation (FAPESP), process number 22/11419-1 and 2021/06285-3.

The data that support the findings of this study are available from the corresponding author upon reasonable request.