The reinstrumentalization of the Diagnostic and Statistical Manual of Mental Disorders (DSM) in psychological publications: A citation context analysis

https://publons.com/publon/10.1162/qss_a_00124

Research instruments (i.e., objects that are instrumental to scientific works), are an important class of material objects involved in scientific research. Studies on scientific practices have found that instruments serve as a fundamental device through which researchers can gain access to “nature” that is otherwise invisible to human beings (Fraassen, 2008; Rheinberger, 1997) and as an important epistemological foundation of scientific objectivity (Daston & Galison, 2010). As quantitative researchers strive to understand the scientific system from broader perspectives (Leydesdorff, Ràfols, & Milojevic, 2020), the representation of scientific material objects, especially instruments, in scientific citations and texts has become an important topic yet to be fully investigated. Specifically, with an increasing number of publications addressing the scientific impact of research data sets and software entities (Howison & Bullard, 2015; Li, Yan, & Feng, 2017; Zhao, Yan, & Li, 2018), two key examples of scientific instruments, we need deeper knowledge about how the use and development of these research instruments are situated in epistemic cultures of science (Knorr-Cetina, 1999).

To address this gap, this paper presents an analysis of the citation contexts of a classic research instrument in the scholarship of mental disorder, the Diagnostic and Statistical Manual of Mental Disorders (DSM), as a response to the concept of instrumentalization in the field of science and technology studies (STS). Works by Bruno Latour and his colleagues (Latour, 1987; Latour & Woolgar, 1979) successfully established a materialist and practice-oriented tradition in science studies. In this line of research, a frequently recurring theme is that the status of research instruments is contingent and dependent on specific research contexts in scientific laboratories that are both temporal and local. Two major arguments have been proposed: that there are highly blurry and fluid boundaries between research instruments and other types of research objects (Engeström, 1990; Rheinberger, 1997) and that scientific knowledge and instruments are always coproduced (Jasanoff, 2004). These ideas are well summarized by Clarke and Fujimura (1992), who famously stated that scientific instruments are constructed through stabilization of scientific knowledge: A knowledge object becomes a tool when it is “no longer questioned, examined, or viewed as problematic, but is taken for granted” (pp. 10–11).

Given this conceptualization of instrumentalization, this study aims to examine how the DSM gained the status of a well-accepted research instrument in the research domain of psychology. Developed by the American Psychiatric Association (APA), the DSM is one of the most widely used classification systems for mental disorders. Originally designed as a tool for interhospital communication in the 1950s, the DSM was gradually developed into a diagnostic scheme used by “psychiatrists, other physicians, and other mental health professionals that described the essential features of the full range of mental disorders” (American Psychiatric Association, 2013). Bowker and Star (2000) offered the observation that the DSM serves as the lingua franca for medical insurance companies, because of a lack of competing standards in the market. Moreover, the DSM is publicly regarded as an authoritative resource for both research and teaching in such fields as psychiatry and psychology, especially in North America (Millon & Klerman, 1986; Young, 1997).

In this study, we aim to examine the instrumentality of DSM in a different space other than scientific laboratories: scientific publications. We assume that instrumentalization of DSM also happened in scientific publications after this instrument was developed, even though this process is rather different from how DSM was physically developed through scientific activities. Inspired by Small’s recent work (Small, 2018), we assume that the references representing research instruments are cited in different citation contexts over their life cycles after being published. As this process happens after the development of the instrument, we call it “reinstrumentalization.” Comparing with the original concept of instrumentalization, reinstrumentalization is also a temporal process during which the cited or mentioned instrument is differently regarded or adopted as a valid tool: The more instrumentalized a tool’s representation is, the more it is used as a tool, instead of a conceptual entity, in publications. However, due to the fact that these differences are inevitably represented as textual features in publications and the diversity of language use in research, reinstrumentalization is expressed as a continuous scale. That is to say, using all scientific publications, we will be able to understand the level of reinstrumentalization of a tool from the perspective of the extent to which it is mentioned within certain linguistic contexts.

According to Clarke and Fujimura (1992), temporality is an important scale along which research objects show instrumentality. This is also the focus of the present study: We aim to understand how the level of re-instrumentality of the DSM has changed over its citation histories. The plural form of history is used here because there are multiple citation histories of the DSM: while we can take the DSM as a single object being cited in the scientific literature, each of its different versions (as shown in Table 1) may be deemed to have its own citation history.

Based on the discussions above, this work aims to establish the temporal connections between the level of re-instrumentality and various linguistic features along with citations to DSM versions. This research is deeply rooted in the tradition of citation context analysis. First proposed in the early 1980s (Small, 1982), citation context analysis deals with the “particular message or statement within the citing document containing the reference” (p. 288), so that deeper meanings of citations can be extracted from publications. This method has frequently been adopted by researchers in quantitative science studies dating back to the 1980s, from Garfield’s analysis of how Robert Merton’s works are cited in different knowledge domains (Garfield, 1980) to McCain’s Mean Utility Index, which considered both the location and context of citations (McCain & Turner, 1989), and her study of the longitudinal citation contexts of Frederick Brooks’ book The Mythical Man-Month (McCain & Salvucci, 2006).

Moreover, following Small’s work (Small, 2018), we likewise assume that more frequent use of the DSM (or a specific DSM version) in the Method section of research articles demonstrates a concomitantly higher level of re-instrumentality. Based on this assumption, we have extended the research framework adopted by Small (2018). We specifically examined the five metadiscoursal resources proposed by Hyland (2005a) as well as verbs used in citation sentences (more details of these measurements are discussed in Section 2.2) from more than 100,000 full-text psychology research articles included in the Elsevier Text and Data Mining service. We hope our study is a first step towards a more profound appreciation of our scholarly communication system from a material-oriented perspective, which will help us shift away from document-centric bias and construct a fairer reward system for all kinds of scholarly outputs.

To conduct this analysis, we acquired all English research articles in psychological journals included in the Elsevier Text and Data Mining (TDM) service¹, Elsevier’s official API service offering access to all content in the Elsevier ScienceDirect full-text database. This collection includes both open-access and institution-subscribed articles. Up to 2018, the ScienceDirect platform has the full text of over 15 million publications from more than 20,000 journals published as early as 1823². This research is conducted on the full Elsevier TDM data set as collected by the Centre for Science and Technology Studies (CWTS) at Leiden University. Using this database, the following data collection steps were undertaken.

First, all 203 journals classified under psychology were acquired on April 15, 2019. All English research articles published in these journals, as collected by CWTS, were included in our sample. We retrieved 203,803 unique DOIs from this step.

Second, from the above collection, we selected only the 145,892 papers with structured full-text data for inclusion in the final sample.

Third, we included only papers with at least one Method section. The following four criteria were used to classify paper sections as either Method or non-Method:

• Criterion #1: A Method section title should contain the following terms: “experimental,” “methodology,” “methodologies,” “method,” or “methods”;

• Criterion #2 A Method section title must not contain the terms “experiment” or “experiments”;

• Criterion #3: Any subsection under a Method section is a Method section; and

• Criterion #4: A Method section can be a subsection of a non-Method section.

This list of criteria considers the fact that psychological papers sometimes contain parallel sections with titles such as “Experiment 1” and “Experiment 2.” These sections serve as the container for a separate set of Method, Results, and sometimes Discussion sections. We tested some additional keywords in our query, especially “data,” but given that these failed to produce any significant increase in the number of retrieved paper sections, we maintained our original query.

This approach to classifying paper sections was tested on the full paper sample and was evaluated manually. After it was applied on our paper sample, we used all 22 Method-section titles that were manually validated and appeared at least 100 times in the sample. All the selected Method-section titles are listed in  Appendix A. These titles have 105,220 instances (in 104,094 unique papers); both figures are comparable to the 119,166 instances of the Introduction section found in our sample. All 104,094 articles were thus included in our final sample for this analysis. These papers were published between 1997 and 2016. A more detailed description of the paper sample is offered in Section 2.4. Based on this sample, we acquired all citances parsed by CWTS from their database.

Fourth, based on our final sample, we identified all references related to the DSM. In the CWTS database, a reference key is given to every unique indexed reference. This key is composed of the name of the leading contributor, the publication year, and some other identifying information (such as the first few characters of the object title). For example, one key for the fifth edition of the DSM is “americanpsychiatricassoci_2013_the.” However, nonpublication objects are frequently cited with variant titles and other metadata elements (Li, Chen, & Yan, 2019), which explains why multiple keys (e.g. “americanpsychiatricassoci_2013_ame”) were found for most DSM versions. To address this issue, we used the following criteria to integrate keys representing the same DSM version.

#1: The author of the reference is the American Psychiatric Association (“americanpsychiatricassoci”);

#2: The object must be published in a year in which a major DSM version was published (as in Table 1);

#3: The initial letters of the title must be either “the,” “dia,” “dsm,” or “ame,” so that it is possible to cover different name forms of the DSM without including other resources published by APA in the same years. The selected letters cover such titles as “The Diagnostic and Statistical Manual of Mental Disorder” and “Diagnostic and Statistical Manual of Mental Disorder.”

In total, 87 unique keys were identified for the seven major versions of the DSM. It should be noted that there are a few other keys that meet criteria #1 and #3, but not #2 (i.e., keys with a different year from those in Table 1). These may be correct references that were given a wrong publication year, mistakenly or intendedly. However, we decided not to use them because of the difficulty of accurately classifying them into the seven versions.

Based on the sample discussed above, the following measurements were examined in this study.

It has been commonly accepted that different sections within a paper have distinct narrative functions and significance (Swales, 1990). An assumption made in this study is that citations in the Method section are more strongly connected to scientific instrumentality; this assumption is supported by various prior works in quantitative science studies (Bertin, Atanassova et al., 2016; Thelwall, 2019) and applied linguistics (Huang, 2014; Kanoksilapatham, 2012). To calculate this parameter, we measured the ratio of DSM citations used in the Method section to all DSM citations in a paper. By tracing this ratio and especially how it changes over time, we strive to illustrate the extent to which a specific version of the DSM is regarded as an established research instrument, how this pattern changes over time, and how well this temporal trend is correlated with other linguistic attributes of DSM citances.

A large number of studies focus on the meanings of a reference when it is cited multiple times in another article. A number of earlier studies reported evidence that the more times a reference is cited in a publication, there is a stronger and more meaningful relationship between the citing and cited documents (Chubin & Moitra, 1975; Hooten, 1991; Tang & Safer, 2008; Voos & Dagaev, 1976). Specifically, this parameter can be an even more effective predictor of citation functions when it is combined with the location of citation, as indicated by multiple studies (McCain & Turner, 1989; Zhao, Cappello, & Johnston, 2017; Zhao & Strotmann, 2020). However, most of these studies were conducted on all research articles and their findings have not been examined in the context of cited research instruments.

Linguistic attributes have been increasingly studied as a type of citation context (Ding, Liu et al., 2013; Jha, Jbara et al., 2017; Small, 2018). In this study, we focused on five classes of interactional markers identified by Hyland (1999): hedges, boosters, attitude markers, self-mentions, and engagement markers. They are, according to Hyland, resources available to writers to interact with their readers through their writings—for instance, in expressing their views or acknowledging uncertainties. Hedges are the only category that has been deliberately examined in quantitative studies. As a result, the present study aims to offer a more comprehensive analysis of how these resources are used in the textual description of scientific instruments. The five resources are discussed below and examples of how these resources are used in our sample are given in  Appendix B.

• Hedges: Hedges, such as possible and perhaps, are devices indicating uncertainties in writing (e.g., in acknowledging an alternative interpretation). Earlier quantitative studies have proven that hedges are inversely related to the Method section and method-oriented references, because of the certain tone that is supposed to be expressed in this section (Chen, Song, & Heo, 2018; Small, 2018; Small, Boyack, & Klavans, 2019).

• Boosters: Contrary to hedges, boosters, such as obviously and demonstrate, are used to express certainties. Despite this opposition of function, Hyland (2005b) commented that the excessive use of either type of marker is discouraged by the research community.

• Attitude markers: Attitude markers are those words or phrases that express the writer’s subjective attitudes towards the topics. Examples include attitude verbs (agree), sentence adverbs (unfortunately), and adjectives (remarkable). While no research has focused on their usage in scientific writings, we can assume that attitudes, like uncertainties, are less likely to be expressed in the Method section than in other paper sections.

• Self-mention: Self-mention includes first-person pronouns (we) and possessive adjectives (our). In an earlier study, Hyland (2003) interpreted the ratios of self-mention in scientific publications across multiple research fields using the different research and persuasion strategies employed in these communities. For example, in the hard sciences, uniformities in the procedures and results are more important in convincing readers than is personal authority; thus, the texts in these fields are less personal in style than in the soft sciences. Despite such disciplinary differences, there is a lack of discussion as to how self-mention phrases are used differently between paper sections.

• Engagement markers: Engagement markers are devices to guide readers’ attention or include them in the discourse. Such expressions include by the way, you may notice, and note.

Readability is a frequently used linguistic attribute in evaluating the writing styles of scientific publications (Hartley, Pennebaker, & Fox, 2003; Hayden, 2008). Despite recent criticism (Hartley, 2016), we used the Flesch Scale in this study, as it is still widely used in quantitative studies of texts and especially scientific texts, as a marker for the differences between individual texts (Didegah, Bowman, & Holmberg, 2018; Oleinik, Kirdina-Chandler et al., 2017; Van Wesel, Wyatt, & ten Haaf, 2014).

Verbs bear strong rhetorical functions in the construction of scientific texts and are especially useful for understanding the relationship between citing and cited documents (Bertin et al., 2016; Bloch, 2010). Several studies have evaluated which verbs are the most frequently used in the Method section (Bertin & Atanassova, 2014; Lamers, van Eck et al., 2018; Small, 2018).

Using the 87 keys mentioned above, we acquired all citances of the DSM and conducted the following analyses.

For the linguistic analysis, we used the Spacy library in Python (Honnibal & Montani, 2017) to parse the sentences and extract verbs from them. Based on the statistics reported on its website, Spacy’s parser reaches an accuracy of 94.48% as tested on a Wall Street Journal data set³. Moreover, this library has been increasingly used to examine scientific corpora in recent years (Lamurias & Couto, 2019; Roth & Basov, 2020). Verbs were extracted based on the part-of-speech tagger implemented in the library. In this study, we extracted and analyzed only the primary verbs in citances, or the verbs in the main clause. If there were multiple main verbs in the same sentence, we included all of them.

To determine whether a sentence used any of the five interactional resources identified by Hyland, we used the list of phrases offered in his book (Hyland, 2005a). All the included phrases were matched with the citances described above.

We used the Flesch Reading Ease Scale to calculate the readability of each citance. This domain-independent readability scale was developed by Rudolf Franz Flesch in 1943 (Flesch, 1948) and considers sentence length and the number of syllables to calculate the ease of reading for a corpus. It assigns a score from 0 to 100 to a corpus, with 100 representing the easiest and 0 the most difficult. The textstat Python package⁴ was used to calculate this parameter.

In aggregate, the identified keys representing the DSM are cited in 17,695 citances belonging to 12,435 papers. Figure 1 summarizes the ratio of citing articles among all sampled articles (n = 145,892) in a given year, and thus the relative importance of the DSM in our general paper sample. Despite the growing numbers of citations over time, the results show that the DSM has been cited in a relatively stable proportion of papers over the citation window.

We also broke up the total number of citing publications by DSM version. The result is shown in Figure 2, with the y-axis representing the ratio of papers citing each version among the total number of papers in our overall sample. We used the object history (the year difference between a version’s publication date and the citation date) as the x-axis, so that the patterns for all DSM versions can be standardized against their different relationships with the citation window. In this graph, the four most recent DSM versions are highlighted. The three other versions (Versions 1–3) are also shown in the graph, but their names are not printed out.

In our research design, only two DSM versions are fully covered by the citation window (from 1997 to 2016): Version 4-TR (2000) and Version 5 (2013). Moreover, Version 4 was published just ahead of the citation window (1994). We can see that each of these three versions is increasingly cited in its early years, despite the very different slopes. After a new version is published, fewer citations are given to older versions. This can be explained by the fact that up-to-dateness is an important factor for researchers in selecting an instrument such as the DSM.

A major focus of this study is to understand how linguistic attributes are used differently in citances over the histories of the DSM. Table 2 summarizes the measurements discussed above. In this table, as well as the rest of the study, we included only versions from V3-TR to V5 because of the small numbers of citations the first three DSM versions received (17, 29, and 399, respectively). The table shows that some measurements have strong variances among these four versions, which is the starting point of this study.

One of the most notable differences among the DSM versions in Table 2 is the ratio of citances used in the Method section. The likelihood for a citation to be given in this section decreased sharply from Version 3-TR to Version 5, as shown in Figure 3, where each data point is an aggregated ratio for a specific year. (This figure omits data points where a version has fewer than 10 citances in a year, to reduce radical outliers.)

Given the fact that these four versions have distinct relationships with the citation window, we further evaluated whether the differences were caused by differences in the portion of the life cycle covered by the citation window. In Figure 4, we plotted how the Method-section citance ratio changed over each version’s citation history. Despite the fact that our citation window does not fully cover most of the DSM versions, there is a similar upward trend in V4 and V4-TR during their fifth to 15th years after publication. This may suggest a similar trend to be observed in V5 during the next few years, given its similar pattern with V4-TR in its earliest citation history. To a lesser extent, this may also indicate how V3-TR was cited before the 1980s, assuming the general citation pattern of DSM has remained the same since the 1970s. Overall, this result is an indication that the DSM, as a well-developed scientific instrument, still takes time to be regarded as an instrument that can be reliably used in scientific research in the space of scientific publications.

Another pattern with a strong linear trend among various versions is the number of citances per paper. As shown in Figure 5, from versions 3-TR to 5, the mean number of citances per paper keeps increasing, while that in the Method section decreases significantly.

Figure 6 shows how both trends change over the citation histories of all DSM versions. They are once again associated with the different life cycles of these versions. In its beginning years, a new version is mentioned in non-Method sections more frequently but barely mentioned in the Method section. In light of the results from the previous section, the decreasing number of citances in non-Method sections in the earlier years may be explained by the need for researchers to introduce or justify the DSM as a research instrument after it is published. This need, of course, gradually diminishes as the DSM is increasingly accepted as an established instrument.

We further examined the linguistic attributes related to DSM citances. Figure 7 illustrates how the five interactional markers are used differently over the citation histories of DSM versions.

As shown in Figure 7, hedges and attention markers are the only two categories with linear changes over the citation history. The decreasing trend in both variables is consistent with our overall assumption that, as the DSM is more established, it is increasingly used as an instrument and thus is described with a lower level of uncertainty (decreasing use of hedges) and in more factual tones (decreasing use of attention markers). The trends for self-mention markers and boosters are essentially flat, and that for engagement markers appears to be subject to radical fluctuation, even though we removed all data points with fewer than 10 citances.

On the other side of the story, some variables examined above display quite different usage patterns in Method and non-Method sections, respectively, even though they may or may not show any temporal trends. The results are summarized in Table 3, where the “All versions” category contains only versions from V3-TR onward, instead of all seven versions of DSM.

A few observations can be drawn from this table. First, both hedges and attention markers are used differently across paper sections and over time. Given the fact that there is a higher ratio of citations being given in the Method section as a reference grows older, this result is consistent with our findings in Figure 7. On average, a citance in a non-Method section is almost three times more likely to have a hedging phrase and four times more likely to have an attention marker than one in the Method section, which makes these features strong predictors of the citation function of a reference. Moreover, there is an especially high use of hedges in citances of the DSM-V5 in the Method section (15.1%), showing that even though it is in the Method section, there is still a higher level of uncertainty being expressed in the very early years of this version.

Second, like hedging and attention markers, boosters, and the readability score exhibit different patterns in the Method section than in other paper sections. Specifically, boosters are also used more frequently in non-Method sections; this is consistent with Hyland’s comment that even in the Method section, the expression of certainty is discouraged, even though it is not consistent with our findings regarding the temporal trend.

The 10 most frequently used verbs extracted from all citances, Method section citances, and non-Method section citances are summarized in Table 4.

It can be observed that our lists of verbs are very different from the top verbs reported in other similar works (Bertin & Atanassova, 2014; Small, Tseng, & Patek, 2017), where verbs that are most frequently used in the Method section normally include use, perform, follow, etc. Most of these verbs rank very low in our results. Moreover, many verbs in our lists do not fall into the category of research verbs (i.e., verbs aiming to describe the procedure or acts of research). However, the top verbs identified from this study are consistent with the distinct ways in which the DSM is involved in research: It is a standard used to diagnose a mental disorder and to support decisions about recruiting participants.

Notwithstanding the differences between our verb lists and those of previous works, we found that verbs could function as a valid means of measuring the text similarity of citances between the Method and non-Method sections. We calculated the frequencies of all 153 verbs that appeared in our sample at least 10 times (with “be” and “have” removed because they are used too broadly). To measure the similarity between these two sections, we applied Spearman rank correlation to the ranks of verbs from the two corpora. Table 5 summarizes the relationships between the three broad citance groups: those in both the Method and non-Method sections, those in the Method section only, and those in the non-Method sections only. All values in the table are significant at the p < 0.05 level. The result shows that the rankings of verbs in the Method and non-Method sections are strongly negatively correlated with each other.

Moreover, we also compared how verbs from each version of the DSM are distributed relative to the abovementioned verb lists. The results, summarized in Table 6, show that verb distributions in newer DSM versions are more similar to those in non-Method sections and vice versa, with all values significant at the p < 0.05 level. This strongly supports our earlier observation that DSM versions are increasingly used in the Method section by showing that the use of verbs can be reliably used to evaluate the instrumentality of references, even though the use of individual verbs may not work very well because of the low density of these verbs in citances.

The present research offers a case study of how citation contexts shift over the citation histories of the DSM in the field of psychology. By conducting a citation context analysis, we examined the relationship between the instrumentality of the DSM and various linguistic attributes, such as the use of interactional markers and verbs in the citation sentences.

The most interesting finding from this research is the fact that, even though every version of the DSM was meticulously developed into a blackboxed instrument (Regier, Narrow et al., 2009), it does not automatically become one after being published, at least in the space of scientific publications. Instead, it still takes time for the DSM to be accepted as a valid instrument by researchers. This is primarily shown by the fact that as a new DSM version matures, it is increasingly used in the Method section compared to its earlier years. This general conclusion is supported by existing works on the rhetorical functions of paper sections (Bertin et al., 2016; Huang, 2014; Kanoksilapatham, 2012) and is also consistent with the citation contexts of sentences in different paper sections, such as the increasing use of DSM citations in the Method section and the opposite trend for the use of hedges and attention markers in DSM citances. The fact that these patterns are shown for all major DSM versions adds further validity to our conclusion.

This conclusion sheds light on the processes of the construction of scientific instruments in two significant ways. First, it aims to bridge the gap between quantitative and qualitative science studies on the topic of scientific instruments. This gap was largely created by various barriers to proper and sufficient representations of research instruments in citation data (Li et al., 2017, 2019). As a result, quantitative researchers have only gathered very limited evidence about the performance of these material objects in the scholarly communication system, even less so from the perspective of their life cycles. This reality makes it very difficult for a material-oriented perspective to be established in quantitative science studies and thus for more conversations to transpire between these two research communities. Second, our evidence shows that, after its physical development and publication, the DSM does not automatically become a research instrument. This process of reinstrumentalization is not covered by science and technology studies literature concerning how research instruments are produced, which shows how quantitative evidence can help to expand theories developed in qualitative communities.

Moreover, in our discussion of the process of reinstrumentalization, we have also shown the significant roles played by the temporal framework in the appreciation of the citation context—in particular, how multiple time frames may exist in an object’s citation history. The concept of version is a vital yet highly underexamined topic in quantitative science studies; versions are critical to the identification of many nonpublication objects, such as research data sets (Pröll & Rauber, 2013) and software objects (Smith, Katz, & Niemeyer, 2016). Moreover, versions are gaining in relevance as prepublication paper repositories are increasingly used by researchers from nearly every field, which creates multiple versions of research articles (Larivière, Sugimoto et al., 2014). This study offers some preliminary evidence about the roles played by versioning in the citation history of an object. Specifically, we have shown that each version of the DSM can be treated as a unique epistemic object, with its own citation history and similar patterns of citation contexts. However, to understand the concept of version in scholarly communication in more depth, we plan to conduct more studies in the future to elucidate how it may work differently for other types of research object and in different knowledge domains.

Another major contribution of the present work is to offer a more comprehensive examination of the relationships between citation contexts and method-related rhetorical functions, inspired by recent work concerning citation contexts of method papers⁵ (Small, 2018; Small et al., 2019). Our results supported Small’s key finding that hedges are a central predictor for method-related citation context, especially the distinctively low level of uncertainties expressed in the Method section. However, apart from hedges, we also evaluated how other interactional phrases are used along the life cycle of the DSM. We found that attention markers are another strong indicator for how the DSM is cited. Like hedges, these show strong variances not only between the Method and non-Method sections, but also between different stages of the DSM’s citation histories. In addition to hedges and attention markers, boosters are shown to be used differently between the Method section and other paper sections. However, this difference does not translate into temporal patterns. Instead, the use of boosters is relatively stable over the citation histories of the DSM.

The different patterns for these interactional markers point to the complexity of human language in scientific writings. One example of such complexity is that citations to the DSM are not consistently given in an instrumental context. This may be applicable to other research instruments, given the diversity of citation practices for material objects (Li et al., 2019). To address this complexity, these linguistic attributes can be employed as useful instruments for future works to automatically identify research instruments from scientific citations and texts, which will further help to construct a material-oriented history of science.

This study also evaluated the distribution of main verbs in all citances of the DSM. Two conclusions are drawn from the analysis. First, the verb profile of the DSM (even when based on all Method section citances) is very different from earlier findings based on all citations in the Method section (Bertin et al., 2016). This finding indicates that even under the citation contexts of the Method section, there are different and highly individual connections between individual citations (or instruments) and verbs (or actions taken in the research). This is consistent with how the Method section is conceptualized in the Create a Research Space (CARS) model, where the section serves to describe a localized research setting (Swales, 1990).

This individualized connection between verbs and citations raises questions about the extent to which we can use a general verb ranking or one scheme of citation context to analyze the Method section, given the vast diversity of research actions taken by researchers. Before fully transforming verbs into a valid research instrument, we need to build better knowledge about the categorization of research actions expressed by action verbs, with a comprehensive consideration of local factors, such as disciplinarity and the life cycle of the research objects represented by the citation. This research direction will be an important complement to existing works on full-text scientific publications.

Despite the individual nature of the connections between verbs and citations, verbs have proven to be a solid instrument to evaluate citation contexts at a somewhat aggregated level. By analyzing the similarity of verb profiles, we draw the same conclusion that older DSM versions are more strongly connected to the Method section than are newer ones.

The research reported in this paper offers a citation context analysis of how the DSM is cited in full-text psychological publications. Our results show that the re-instrumentality, or the extent to which the DSM is used in the Method section, varies as specific versions of the DSM go through their respective life stages. Over the first few years after its publication, a DSM version is increasingly cited in the Method section, which indicates that it is increasingly cited as a research instrument over time. This ratio does not seem to be influenced by the publication of the next version and only reaches its peak after around 10–15 years. Moreover, this changing level of instrumentality is accompanied by shifts in the use of some other important citation contexts within citation sentences. We found significantly different uses of hedges, attention markers, and some verbs as the DSM becomes more mature.

Our study offers an important quantitative examination of how citations serve as research instruments, one citation function that has not been extensively studied in quantitative science studies. Specifically, this research builds connections between quantitative evidence and theories of instrumentalization developed in qualitative studies in the field of science and technology studies. We found another aspect of how research instruments are constructed that is not fully addressed in the qualitative literature: After “physical” packaging, research instruments need to be re-instrumentalized in the space of scientific texts, as reflected in those texts and their citations.

This paper, despite the significance of its findings, represents only the first step towards a deeper understanding of how research instruments are represented in scientific texts. After all, it is only a case study of a unique research instrument on many different levels. To address this limitation, as the next step of this research project, we plan to conduct larger-scale quantitative analyses to examine the findings from the present study more comprehensively, especially the manner in which these research instruments are used differently across knowledge domains. Moreover, it is also possible that DSM is mentioned, instead of cited, in these publications, as DSM versions are “obliterated” into mere mentions over their citation histories (McCain, 2014). While this question cannot be investigated by this current study, it is an important question to be pursued in future works.

Part of this work was done while the author was a visiting researcher at the Centre for Science and Technology Studies (CWTS) at Leiden University. The author thanks CWTS for providing access to its data infrastructure. The author would also like to thank Drs. Erjia Yan, Cassidy Sugimoto, Chaomei Chen, Jake Williams, Jane Greenberg, and Ali Kenner for their helps in the composition of this article. My thanks also go to Dr. Ludo Waltman and peer reviewers for their comments on the manuscript.

No funding was received for this research.

The author has no competing interests.

The data cannot be made available in a data repository because the provider of the data (Elsevier) does not allow this.