Abstract

Research instruments play significant roles in the construction of scientific knowledge, even though we have only acquired very limited knowledge about their life cycles from quantitative studies. This paper aims to address this gap by quantitatively examining the citation contexts of an exemplary research instrument, the Diagnostic and Statistical Manual of Mental Disorders (DSM), in full-text psychological publications. We investigated the relationship between the citation contexts of the DSM and its status as a valid instrument being used and described by psychological researchers. We specifically focused on how this relationship has changed over the DSM’s citation histories, especially through the temporal framework of its versions. We found that a new version of the DSM is increasingly regarded as a valid instrument after its publication; this is reflected in various key citation contexts, such as the use of hedges, attention markers, and the verb profile in sentences where the DSM is cited. We call this process the reinstrumentalization of the DSM in the space of scientific publications. Our findings bridge an important gap between quantitative and qualitative science studies and shed light on an aspect of the social process of scientific instrument development that is not addressed by the current qualitative literature.

PEER REVIEW

1. INTRODUCTION

Research instruments (i.e., objects that are instrumental to scientific works), are an important class of material objects involved in scientific research. Studies on scientific practices have found that instruments serve as a fundamental device through which researchers can gain access to “nature” that is otherwise invisible to human beings (Fraassen, 2008; Rheinberger, 1997) and as an important epistemological foundation of scientific objectivity (Daston & Galison, 2010). As quantitative researchers strive to understand the scientific system from broader perspectives (Leydesdorff, Ràfols, & Milojevic, 2020), the representation of scientific material objects, especially instruments, in scientific citations and texts has become an important topic yet to be fully investigated. Specifically, with an increasing number of publications addressing the scientific impact of research data sets and software entities (Howison & Bullard, 2015; Li, Yan, & Feng, 2017; Zhao, Yan, & Li, 2018), two key examples of scientific instruments, we need deeper knowledge about how the use and development of these research instruments are situated in epistemic cultures of science (Knorr-Cetina, 1999).

To address this gap, this paper presents an analysis of the citation contexts of a classic research instrument in the scholarship of mental disorder, the Diagnostic and Statistical Manual of Mental Disorders (DSM), as a response to the concept of instrumentalization in the field of science and technology studies (STS). Works by Bruno Latour and his colleagues (Latour, 1987; Latour & Woolgar, 1979) successfully established a materialist and practice-oriented tradition in science studies. In this line of research, a frequently recurring theme is that the status of research instruments is contingent and dependent on specific research contexts in scientific laboratories that are both temporal and local. Two major arguments have been proposed: that there are highly blurry and fluid boundaries between research instruments and other types of research objects (Engeström, 1990; Rheinberger, 1997) and that scientific knowledge and instruments are always coproduced (Jasanoff, 2004). These ideas are well summarized by Clarke and Fujimura (1992), who famously stated that scientific instruments are constructed through stabilization of scientific knowledge: A knowledge object becomes a tool when it is “no longer questioned, examined, or viewed as problematic, but is taken for granted” (pp. 10–11).

Given this conceptualization of instrumentalization, this study aims to examine how the DSM gained the status of a well-accepted research instrument in the research domain of psychology. Developed by the American Psychiatric Association (APA), the DSM is one of the most widely used classification systems for mental disorders. Originally designed as a tool for interhospital communication in the 1950s, the DSM was gradually developed into a diagnostic scheme used by “psychiatrists, other physicians, and other mental health professionals that described the essential features of the full range of mental disorders” (American Psychiatric Association, 2013). Bowker and Star (2000) offered the observation that the DSM serves as the lingua franca for medical insurance companies, because of a lack of competing standards in the market. Moreover, the DSM is publicly regarded as an authoritative resource for both research and teaching in such fields as psychiatry and psychology, especially in North America (Millon & Klerman, 1986; Young, 1997).

In this study, we aim to examine the instrumentality of DSM in a different space other than scientific laboratories: scientific publications. We assume that instrumentalization of DSM also happened in scientific publications after this instrument was developed, even though this process is rather different from how DSM was physically developed through scientific activities. Inspired by Small’s recent work (Small, 2018), we assume that the references representing research instruments are cited in different citation contexts over their life cycles after being published. As this process happens after the development of the instrument, we call it “reinstrumentalization.” Comparing with the original concept of instrumentalization, reinstrumentalization is also a temporal process during which the cited or mentioned instrument is differently regarded or adopted as a valid tool: The more instrumentalized a tool’s representation is, the more it is used as a tool, instead of a conceptual entity, in publications. However, due to the fact that these differences are inevitably represented as textual features in publications and the diversity of language use in research, reinstrumentalization is expressed as a continuous scale. That is to say, using all scientific publications, we will be able to understand the level of reinstrumentalization of a tool from the perspective of the extent to which it is mentioned within certain linguistic contexts.

According to Clarke and Fujimura (1992), temporality is an important scale along which research objects show instrumentality. This is also the focus of the present study: We aim to understand how the level of re-instrumentality of the DSM has changed over its citation histories. The plural form of history is used here because there are multiple citation histories of the DSM: while we can take the DSM as a single object being cited in the scientific literature, each of its different versions (as shown in Table 1) may be deemed to have its own citation history.

Table 1.

Versions of the DSM

VersionAbbreviationPublication year
1st Edition 1952 
2nd Edition 1968 
3rd Edition 1980 
3rd Edition (Text Revision) 3-TR 1987 
4th Edition 1994 
4th Edition (Text Revision) 4-TR 2000 
5th Edition 2013 
VersionAbbreviationPublication year
1st Edition 1952 
2nd Edition 1968 
3rd Edition 1980 
3rd Edition (Text Revision) 3-TR 1987 
4th Edition 1994 
4th Edition (Text Revision) 4-TR 2000 
5th Edition 2013 

Based on the discussions above, this work aims to establish the temporal connections between the level of re-instrumentality and various linguistic features along with citations to DSM versions. This research is deeply rooted in the tradition of citation context analysis. First proposed in the early 1980s (Small, 1982), citation context analysis deals with the “particular message or statement within the citing document containing the reference” (p. 288), so that deeper meanings of citations can be extracted from publications. This method has frequently been adopted by researchers in quantitative science studies dating back to the 1980s, from Garfield’s analysis of how Robert Merton’s works are cited in different knowledge domains (Garfield, 1980) to McCain’s Mean Utility Index, which considered both the location and context of citations (McCain & Turner, 1989), and her study of the longitudinal citation contexts of Frederick Brooks’ book The Mythical Man-Month (McCain & Salvucci, 2006).

Moreover, following Small’s work (Small, 2018), we likewise assume that more frequent use of the DSM (or a specific DSM version) in the Method section of research articles demonstrates a concomitantly higher level of re-instrumentality. Based on this assumption, we have extended the research framework adopted by Small (2018). We specifically examined the five metadiscoursal resources proposed by Hyland (2005a) as well as verbs used in citation sentences (more details of these measurements are discussed in Section 2.2) from more than 100,000 full-text psychology research articles included in the Elsevier Text and Data Mining service. We hope our study is a first step towards a more profound appreciation of our scholarly communication system from a material-oriented perspective, which will help us shift away from document-centric bias and construct a fairer reward system for all kinds of scholarly outputs.

2. METHOD

2.1. Sample

To conduct this analysis, we acquired all English research articles in psychological journals included in the Elsevier Text and Data Mining (TDM) service1, Elsevier’s official API service offering access to all content in the Elsevier ScienceDirect full-text database. This collection includes both open-access and institution-subscribed articles. Up to 2018, the ScienceDirect platform has the full text of over 15 million publications from more than 20,000 journals published as early as 18232. This research is conducted on the full Elsevier TDM data set as collected by the Centre for Science and Technology Studies (CWTS) at Leiden University. Using this database, the following data collection steps were undertaken.

First, all 203 journals classified under psychology were acquired on April 15, 2019. All English research articles published in these journals, as collected by CWTS, were included in our sample. We retrieved 203,803 unique DOIs from this step.

Second, from the above collection, we selected only the 145,892 papers with structured full-text data for inclusion in the final sample.

Third, we included only papers with at least one Method section. The following four criteria were used to classify paper sections as either Method or non-Method:

  • Criterion #1: A Method section title should contain the following terms: “experimental,” “methodology,” “methodologies,” “method,” or “methods”;

  • Criterion #2 A Method section title must not contain the terms “experiment” or “experiments”;

  • Criterion #3: Any subsection under a Method section is a Method section; and

  • Criterion #4: A Method section can be a subsection of a non-Method section.

This list of criteria considers the fact that psychological papers sometimes contain parallel sections with titles such as “Experiment 1” and “Experiment 2.” These sections serve as the container for a separate set of Method, Results, and sometimes Discussion sections. We tested some additional keywords in our query, especially “data,” but given that these failed to produce any significant increase in the number of retrieved paper sections, we maintained our original query.

This approach to classifying paper sections was tested on the full paper sample and was evaluated manually. After it was applied on our paper sample, we used all 22 Method-section titles that were manually validated and appeared at least 100 times in the sample. All the selected Method-section titles are listed in  Appendix A. These titles have 105,220 instances (in 104,094 unique papers); both figures are comparable to the 119,166 instances of the Introduction section found in our sample. All 104,094 articles were thus included in our final sample for this analysis. These papers were published between 1997 and 2016. A more detailed description of the paper sample is offered in Section 2.4. Based on this sample, we acquired all citances parsed by CWTS from their database.

Fourth, based on our final sample, we identified all references related to the DSM. In the CWTS database, a reference key is given to every unique indexed reference. This key is composed of the name of the leading contributor, the publication year, and some other identifying information (such as the first few characters of the object title). For example, one key for the fifth edition of the DSM is “americanpsychiatricassoci_2013_the.” However, nonpublication objects are frequently cited with variant titles and other metadata elements (Li, Chen, & Yan, 2019), which explains why multiple keys (e.g. “americanpsychiatricassoci_2013_ame”) were found for most DSM versions. To address this issue, we used the following criteria to integrate keys representing the same DSM version.

#1: The author of the reference is the American Psychiatric Association (“americanpsychiatricassoci”);

#2: The object must be published in a year in which a major DSM version was published (as in Table 1);

#3: The initial letters of the title must be either “the,” “dia,” “dsm,” or “ame,” so that it is possible to cover different name forms of the DSM without including other resources published by APA in the same years. The selected letters cover such titles as “The Diagnostic and Statistical Manual of Mental Disorder” and “Diagnostic and Statistical Manual of Mental Disorder.”

In total, 87 unique keys were identified for the seven major versions of the DSM. It should be noted that there are a few other keys that meet criteria #1 and #3, but not #2 (i.e., keys with a different year from those in Table 1). These may be correct references that were given a wrong publication year, mistakenly or intendedly. However, we decided not to use them because of the difficulty of accurately classifying them into the seven versions.

2.2. Measurements

Based on the sample discussed above, the following measurements were examined in this study.

2.2.1. Ratio of citations in the Method section

It has been commonly accepted that different sections within a paper have distinct narrative functions and significance (Swales, 1990). An assumption made in this study is that citations in the Method section are more strongly connected to scientific instrumentality; this assumption is supported by various prior works in quantitative science studies (Bertin, Atanassova et al., 2016; Thelwall, 2019) and applied linguistics (Huang, 2014; Kanoksilapatham, 2012). To calculate this parameter, we measured the ratio of DSM citations used in the Method section to all DSM citations in a paper. By tracing this ratio and especially how it changes over time, we strive to illustrate the extent to which a specific version of the DSM is regarded as an established research instrument, how this pattern changes over time, and how well this temporal trend is correlated with other linguistic attributes of DSM citances.

2.2.2. Number of times a reference is cited in a paper

A large number of studies focus on the meanings of a reference when it is cited multiple times in another article. A number of earlier studies reported evidence that the more times a reference is cited in a publication, there is a stronger and more meaningful relationship between the citing and cited documents (Chubin & Moitra, 1975; Hooten, 1991; Tang & Safer, 2008; Voos & Dagaev, 1976). Specifically, this parameter can be an even more effective predictor of citation functions when it is combined with the location of citation, as indicated by multiple studies (McCain & Turner, 1989; Zhao, Cappello, & Johnston, 2017; Zhao & Strotmann, 2020). However, most of these studies were conducted on all research articles and their findings have not been examined in the context of cited research instruments.

2.2.3. Linguistic attributes of citances

Linguistic attributes have been increasingly studied as a type of citation context (Ding, Liu et al., 2013; Jha, Jbara et al., 2017; Small, 2018). In this study, we focused on five classes of interactional markers identified by Hyland (1999): hedges, boosters, attitude markers, self-mentions, and engagement markers. They are, according to Hyland, resources available to writers to interact with their readers through their writings—for instance, in expressing their views or acknowledging uncertainties. Hedges are the only category that has been deliberately examined in quantitative studies. As a result, the present study aims to offer a more comprehensive analysis of how these resources are used in the textual description of scientific instruments. The five resources are discussed below and examples of how these resources are used in our sample are given in  Appendix B.

  • Hedges: Hedges, such as possible and perhaps, are devices indicating uncertainties in writing (e.g., in acknowledging an alternative interpretation). Earlier quantitative studies have proven that hedges are inversely related to the Method section and method-oriented references, because of the certain tone that is supposed to be expressed in this section (Chen, Song, & Heo, 2018; Small, 2018; Small, Boyack, & Klavans, 2019).

  • Boosters: Contrary to hedges, boosters, such as obviously and demonstrate, are used to express certainties. Despite this opposition of function, Hyland (2005b) commented that the excessive use of either type of marker is discouraged by the research community.

  • Attitude markers: Attitude markers are those words or phrases that express the writer’s subjective attitudes towards the topics. Examples include attitude verbs (agree), sentence adverbs (unfortunately), and adjectives (remarkable). While no research has focused on their usage in scientific writings, we can assume that attitudes, like uncertainties, are less likely to be expressed in the Method section than in other paper sections.

  • Self-mention: Self-mention includes first-person pronouns (we) and possessive adjectives (our). In an earlier study, Hyland (2003) interpreted the ratios of self-mention in scientific publications across multiple research fields using the different research and persuasion strategies employed in these communities. For example, in the hard sciences, uniformities in the procedures and results are more important in convincing readers than is personal authority; thus, the texts in these fields are less personal in style than in the soft sciences. Despite such disciplinary differences, there is a lack of discussion as to how self-mention phrases are used differently between paper sections.

  • Engagement markers: Engagement markers are devices to guide readers’ attention or include them in the discourse. Such expressions include by the way, you may notice, and note.

2.2.4. Readability of citances

Readability is a frequently used linguistic attribute in evaluating the writing styles of scientific publications (Hartley, Pennebaker, & Fox, 2003; Hayden, 2008). Despite recent criticism (Hartley, 2016), we used the Flesch Scale in this study, as it is still widely used in quantitative studies of texts and especially scientific texts, as a marker for the differences between individual texts (Didegah, Bowman, & Holmberg, 2018; Oleinik, Kirdina-Chandler et al., 2017; Van Wesel, Wyatt, & ten Haaf, 2014).

2.2.5. Verbs used in citances

Verbs bear strong rhetorical functions in the construction of scientific texts and are especially useful for understanding the relationship between citing and cited documents (Bertin et al., 2016; Bloch, 2010). Several studies have evaluated which verbs are the most frequently used in the Method section (Bertin & Atanassova, 2014; Lamers, van Eck et al., 2018; Small, 2018).

2.3. Analysis Method

Using the 87 keys mentioned above, we acquired all citances of the DSM and conducted the following analyses.

For the linguistic analysis, we used the Spacy library in Python (Honnibal & Montani, 2017) to parse the sentences and extract verbs from them. Based on the statistics reported on its website, Spacy’s parser reaches an accuracy of 94.48% as tested on a Wall Street Journal data set3. Moreover, this library has been increasingly used to examine scientific corpora in recent years (Lamurias & Couto, 2019; Roth & Basov, 2020). Verbs were extracted based on the part-of-speech tagger implemented in the library. In this study, we extracted and analyzed only the primary verbs in citances, or the verbs in the main clause. If there were multiple main verbs in the same sentence, we included all of them.

To determine whether a sentence used any of the five interactional resources identified by Hyland, we used the list of phrases offered in his book (Hyland, 2005a). All the included phrases were matched with the citances described above.

We used the Flesch Reading Ease Scale to calculate the readability of each citance. This domain-independent readability scale was developed by Rudolf Franz Flesch in 1943 (Flesch, 1948) and considers sentence length and the number of syllables to calculate the ease of reading for a corpus. It assigns a score from 0 to 100 to a corpus, with 100 representing the easiest and 0 the most difficult. The textstat Python package4 was used to calculate this parameter.

2.4. Description of the Sample

In aggregate, the identified keys representing the DSM are cited in 17,695 citances belonging to 12,435 papers. Figure 1 summarizes the ratio of citing articles among all sampled articles (n = 145,892) in a given year, and thus the relative importance of the DSM in our general paper sample. Despite the growing numbers of citations over time, the results show that the DSM has been cited in a relatively stable proportion of papers over the citation window.

Figure 1.

Ratio of articles citing the DSM over time.

Figure 1.

Ratio of articles citing the DSM over time.

We also broke up the total number of citing publications by DSM version. The result is shown in Figure 2, with the y-axis representing the ratio of papers citing each version among the total number of papers in our overall sample. We used the object history (the year difference between a version’s publication date and the citation date) as the x-axis, so that the patterns for all DSM versions can be standardized against their different relationships with the citation window. In this graph, the four most recent DSM versions are highlighted. The three other versions (Versions 1–3) are also shown in the graph, but their names are not printed out.

Figure 2.

Ratios of citing articles citing specific versions of the DSM.

Figure 2.

Ratios of citing articles citing specific versions of the DSM.

In our research design, only two DSM versions are fully covered by the citation window (from 1997 to 2016): Version 4-TR (2000) and Version 5 (2013). Moreover, Version 4 was published just ahead of the citation window (1994). We can see that each of these three versions is increasingly cited in its early years, despite the very different slopes. After a new version is published, fewer citations are given to older versions. This can be explained by the fact that up-to-dateness is an important factor for researchers in selecting an instrument such as the DSM.

A major focus of this study is to understand how linguistic attributes are used differently in citances over the histories of the DSM. Table 2 summarizes the measurements discussed above. In this table, as well as the rest of the study, we included only versions from V3-TR to V5 because of the small numbers of citations the first three DSM versions received (17, 29, and 399, respectively). The table shows that some measurements have strong variances among these four versions, which is the starting point of this study.

Table 2.

Summary of key attributes of DSM versions

 V3-TRV4V4-TRV5
No. of citing papers 980 5419 3890 1709 
Ratio of citing papers with Method-section citance 64.8% 63.7% 53.2% 16.8% 
No. of citances 1143 7630 5725 2702 
No. of citances in Method section 59.8% 52.1% 42.1% 11.8% 
Citances per paper 1.17 1.4 1.47 1.58 
Ratio of citances with attention markers 2.3% 4.4% 5.6% 7.5% 
Ratio of citances with boosters 9.4% 8.7% 9.2% 8.9% 
Ratio of citances with self-mentions 3.4% 4.6% 3.6% 5.2% 
Ratio of citances with engagement markers 21.7% 20.6% 19.9% 20.3% 
Ratio of citances with hedging 10.7% 16.2% 19.6% 26.6% 
Mean readability score 22.62 18.96 15.56 13.86 
 V3-TRV4V4-TRV5
No. of citing papers 980 5419 3890 1709 
Ratio of citing papers with Method-section citance 64.8% 63.7% 53.2% 16.8% 
No. of citances 1143 7630 5725 2702 
No. of citances in Method section 59.8% 52.1% 42.1% 11.8% 
Citances per paper 1.17 1.4 1.47 1.58 
Ratio of citances with attention markers 2.3% 4.4% 5.6% 7.5% 
Ratio of citances with boosters 9.4% 8.7% 9.2% 8.9% 
Ratio of citances with self-mentions 3.4% 4.6% 3.6% 5.2% 
Ratio of citances with engagement markers 21.7% 20.6% 19.9% 20.3% 
Ratio of citances with hedging 10.7% 16.2% 19.6% 26.6% 
Mean readability score 22.62 18.96 15.56 13.86 

3. RESULTS

3.1. How Often Is the DSM Cited in the Method Section?

One of the most notable differences among the DSM versions in Table 2 is the ratio of citances used in the Method section. The likelihood for a citation to be given in this section decreased sharply from Version 3-TR to Version 5, as shown in Figure 3, where each data point is an aggregated ratio for a specific year. (This figure omits data points where a version has fewer than 10 citances in a year, to reduce radical outliers.)

Figure 3.

Ratio of Method section citances by DSM version.

Figure 3.

Ratio of Method section citances by DSM version.

Given the fact that these four versions have distinct relationships with the citation window, we further evaluated whether the differences were caused by differences in the portion of the life cycle covered by the citation window. In Figure 4, we plotted how the Method-section citance ratio changed over each version’s citation history. Despite the fact that our citation window does not fully cover most of the DSM versions, there is a similar upward trend in V4 and V4-TR during their fifth to 15th years after publication. This may suggest a similar trend to be observed in V5 during the next few years, given its similar pattern with V4-TR in its earliest citation history. To a lesser extent, this may also indicate how V3-TR was cited before the 1980s, assuming the general citation pattern of DSM has remained the same since the 1970s. Overall, this result is an indication that the DSM, as a well-developed scientific instrument, still takes time to be regarded as an instrument that can be reliably used in scientific research in the space of scientific publications.

Figure 4.

Ratio of Method section citances over years by DSM version.

Figure 4.

Ratio of Method section citances over years by DSM version.

3.2. Density of Citance in Papers

Another pattern with a strong linear trend among various versions is the number of citances per paper. As shown in Figure 5, from versions 3-TR to 5, the mean number of citances per paper keeps increasing, while that in the Method section decreases significantly.

Figure 5.

Mean number of citances per paper by DSM version.

Figure 5.

Mean number of citances per paper by DSM version.

Figure 6 shows how both trends change over the citation histories of all DSM versions. They are once again associated with the different life cycles of these versions. In its beginning years, a new version is mentioned in non-Method sections more frequently but barely mentioned in the Method section. In light of the results from the previous section, the decreasing number of citances in non-Method sections in the earlier years may be explained by the need for researchers to introduce or justify the DSM as a research instrument after it is published. This need, of course, gradually diminishes as the DSM is increasingly accepted as an established instrument.

Figure 6.

Mean number of citances per paper by DSM history.

Figure 6.

Mean number of citances per paper by DSM history.

3.3. Linguistic Analysis

We further examined the linguistic attributes related to DSM citances. Figure 7 illustrates how the five interactional markers are used differently over the citation histories of DSM versions.

Figure 7.

Mean ratio of citances with Hyland’s interactional markers.

Figure 7.

Mean ratio of citances with Hyland’s interactional markers.

As shown in Figure 7, hedges and attention markers are the only two categories with linear changes over the citation history. The decreasing trend in both variables is consistent with our overall assumption that, as the DSM is more established, it is increasingly used as an instrument and thus is described with a lower level of uncertainty (decreasing use of hedges) and in more factual tones (decreasing use of attention markers). The trends for self-mention markers and boosters are essentially flat, and that for engagement markers appears to be subject to radical fluctuation, even though we removed all data points with fewer than 10 citances.

On the other side of the story, some variables examined above display quite different usage patterns in Method and non-Method sections, respectively, even though they may or may not show any temporal trends. The results are summarized in Table 3, where the “All versions” category contains only versions from V3-TR onward, instead of all seven versions of DSM.

Table 3.

Summary of linguistic attributes by DSM version

VersionSectionAttentionBoostersSelf-mentionEngagementHedgingReadability
V3-TR Method 1.3% 7.4% 3.6% 21.4% 7% 22.77 
V3-TR Non-Method 4.3% 13.4% 3% 22.2% 18.3% 22.3 
V4 Method 1.4% 6% 4.3% 20% 6.9% 22.83 
V4 Non-Method 8.6% 12.5% 4.9% 21.6% 29.4% 13.39 
V4-TR Method 2% 6.6% 4% 20.8% 7.6% 22.11 
V4-TR Non-Method 8.8% 11.5% 3.2% 19.1% 30.2% 9.78 
V5 Method 1.3% 7.5% 9.7% 27.3% 15.1% 28.45 
V5 Non-Method 8.5% 9.2% 4.5% 19.2% 28.4% 11.65 
All versions Method 1.5% 6.4% 4.5% 20.8% 7.5% 22.84 
All versions Non-Method 8.4% 11.3% 4% 20.1% 28.8% 12.25 
VersionSectionAttentionBoostersSelf-mentionEngagementHedgingReadability
V3-TR Method 1.3% 7.4% 3.6% 21.4% 7% 22.77 
V3-TR Non-Method 4.3% 13.4% 3% 22.2% 18.3% 22.3 
V4 Method 1.4% 6% 4.3% 20% 6.9% 22.83 
V4 Non-Method 8.6% 12.5% 4.9% 21.6% 29.4% 13.39 
V4-TR Method 2% 6.6% 4% 20.8% 7.6% 22.11 
V4-TR Non-Method 8.8% 11.5% 3.2% 19.1% 30.2% 9.78 
V5 Method 1.3% 7.5% 9.7% 27.3% 15.1% 28.45 
V5 Non-Method 8.5% 9.2% 4.5% 19.2% 28.4% 11.65 
All versions Method 1.5% 6.4% 4.5% 20.8% 7.5% 22.84 
All versions Non-Method 8.4% 11.3% 4% 20.1% 28.8% 12.25 

A few observations can be drawn from this table. First, both hedges and attention markers are used differently across paper sections and over time. Given the fact that there is a higher ratio of citations being given in the Method section as a reference grows older, this result is consistent with our findings in Figure 7. On average, a citance in a non-Method section is almost three times more likely to have a hedging phrase and four times more likely to have an attention marker than one in the Method section, which makes these features strong predictors of the citation function of a reference. Moreover, there is an especially high use of hedges in citances of the DSM-V5 in the Method section (15.1%), showing that even though it is in the Method section, there is still a higher level of uncertainty being expressed in the very early years of this version.

Second, like hedging and attention markers, boosters, and the readability score exhibit different patterns in the Method section than in other paper sections. Specifically, boosters are also used more frequently in non-Method sections; this is consistent with Hyland’s comment that even in the Method section, the expression of certainty is discouraged, even though it is not consistent with our findings regarding the temporal trend.

3.4. Verbs Used in Citances

The 10 most frequently used verbs extracted from all citances, Method section citances, and non-Method section citances are summarized in Table 4.

Table 4.

Top 10 verbs from our sample

RankAllMethodNon-Method
include include characterize 
meet meet include 
characterize diagnose define 
diagnose assess classify 
use use consider 
assess make report 
define base suggest 
make recruit use 
base consist diagnose 
10 recruit establish find 
RankAllMethodNon-Method
include include characterize 
meet meet include 
characterize diagnose define 
diagnose assess classify 
use use consider 
assess make report 
define base suggest 
make recruit use 
base consist diagnose 
10 recruit establish find 

It can be observed that our lists of verbs are very different from the top verbs reported in other similar works (Bertin & Atanassova, 2014; Small, Tseng, & Patek, 2017), where verbs that are most frequently used in the Method section normally include use, perform, follow, etc. Most of these verbs rank very low in our results. Moreover, many verbs in our lists do not fall into the category of research verbs (i.e., verbs aiming to describe the procedure or acts of research). However, the top verbs identified from this study are consistent with the distinct ways in which the DSM is involved in research: It is a standard used to diagnose a mental disorder and to support decisions about recruiting participants.

Notwithstanding the differences between our verb lists and those of previous works, we found that verbs could function as a valid means of measuring the text similarity of citances between the Method and non-Method sections. We calculated the frequencies of all 153 verbs that appeared in our sample at least 10 times (with “be” and “have” removed because they are used too broadly). To measure the similarity between these two sections, we applied Spearman rank correlation to the ranks of verbs from the two corpora. Table 5 summarizes the relationships between the three broad citance groups: those in both the Method and non-Method sections, those in the Method section only, and those in the non-Method sections only. All values in the table are significant at the p < 0.05 level. The result shows that the rankings of verbs in the Method and non-Method sections are strongly negatively correlated with each other.

Table 5.

Spearman rank correlation among verb lists from Method and non-Method citances

CategoryRho value
All citances—Method citances 0.708 
All citances—Non-Method citances 0.509 
Method citances—Non-Method citances −0.547 
CategoryRho value
All citances—Method citances 0.708 
All citances—Non-Method citances 0.509 
Method citances—Non-Method citances −0.547 

Moreover, we also compared how verbs from each version of the DSM are distributed relative to the abovementioned verb lists. The results, summarized in Table 6, show that verb distributions in newer DSM versions are more similar to those in non-Method sections and vice versa, with all values significant at the p < 0.05 level. This strongly supports our earlier observation that DSM versions are increasingly used in the Method section by showing that the use of verbs can be reliably used to evaluate the instrumentality of references, even though the use of individual verbs may not work very well because of the low density of these verbs in citances.

Table 6.

Spearman rank correlation between verbs in specific versions and general lists

CategoryAll to MethodAll to non-MethodMethod to MethodNon-Method to non-Method
V3-TR 0.707 0.175 0.674 0.502 
V4 0.799 0.298 0.865 0.91 
V4-TR 0.602 0.507 0.774 0.892 
V5 0.030 0.801 0.546 0.877 
CategoryAll to MethodAll to non-MethodMethod to MethodNon-Method to non-Method
V3-TR 0.707 0.175 0.674 0.502 
V4 0.799 0.298 0.865 0.91 
V4-TR 0.602 0.507 0.774 0.892 
V5 0.030 0.801 0.546 0.877 

4. DISCUSSION

4.1. The Reinstrumentalization of the DSM in Scientific Texts

The present research offers a case study of how citation contexts shift over the citation histories of the DSM in the field of psychology. By conducting a citation context analysis, we examined the relationship between the instrumentality of the DSM and various linguistic attributes, such as the use of interactional markers and verbs in the citation sentences.

The most interesting finding from this research is the fact that, even though every version of the DSM was meticulously developed into a blackboxed instrument (Regier, Narrow et al., 2009), it does not automatically become one after being published, at least in the space of scientific publications. Instead, it still takes time for the DSM to be accepted as a valid instrument by researchers. This is primarily shown by the fact that as a new DSM version matures, it is increasingly used in the Method section compared to its earlier years. This general conclusion is supported by existing works on the rhetorical functions of paper sections (Bertin et al., 2016; Huang, 2014; Kanoksilapatham, 2012) and is also consistent with the citation contexts of sentences in different paper sections, such as the increasing use of DSM citations in the Method section and the opposite trend for the use of hedges and attention markers in DSM citances. The fact that these patterns are shown for all major DSM versions adds further validity to our conclusion.

This conclusion sheds light on the processes of the construction of scientific instruments in two significant ways. First, it aims to bridge the gap between quantitative and qualitative science studies on the topic of scientific instruments. This gap was largely created by various barriers to proper and sufficient representations of research instruments in citation data (Li et al., 2017, 2019). As a result, quantitative researchers have only gathered very limited evidence about the performance of these material objects in the scholarly communication system, even less so from the perspective of their life cycles. This reality makes it very difficult for a material-oriented perspective to be established in quantitative science studies and thus for more conversations to transpire between these two research communities. Second, our evidence shows that, after its physical development and publication, the DSM does not automatically become a research instrument. This process of reinstrumentalization is not covered by science and technology studies literature concerning how research instruments are produced, which shows how quantitative evidence can help to expand theories developed in qualitative communities.

Moreover, in our discussion of the process of reinstrumentalization, we have also shown the significant roles played by the temporal framework in the appreciation of the citation context—in particular, how multiple time frames may exist in an object’s citation history. The concept of version is a vital yet highly underexamined topic in quantitative science studies; versions are critical to the identification of many nonpublication objects, such as research data sets (Pröll & Rauber, 2013) and software objects (Smith, Katz, & Niemeyer, 2016). Moreover, versions are gaining in relevance as prepublication paper repositories are increasingly used by researchers from nearly every field, which creates multiple versions of research articles (Larivière, Sugimoto et al., 2014). This study offers some preliminary evidence about the roles played by versioning in the citation history of an object. Specifically, we have shown that each version of the DSM can be treated as a unique epistemic object, with its own citation history and similar patterns of citation contexts. However, to understand the concept of version in scholarly communication in more depth, we plan to conduct more studies in the future to elucidate how it may work differently for other types of research object and in different knowledge domains.

4.2. Citation Contexts of Research Instruments

Another major contribution of the present work is to offer a more comprehensive examination of the relationships between citation contexts and method-related rhetorical functions, inspired by recent work concerning citation contexts of method papers5 (Small, 2018; Small et al., 2019). Our results supported Small’s key finding that hedges are a central predictor for method-related citation context, especially the distinctively low level of uncertainties expressed in the Method section. However, apart from hedges, we also evaluated how other interactional phrases are used along the life cycle of the DSM. We found that attention markers are another strong indicator for how the DSM is cited. Like hedges, these show strong variances not only between the Method and non-Method sections, but also between different stages of the DSM’s citation histories. In addition to hedges and attention markers, boosters are shown to be used differently between the Method section and other paper sections. However, this difference does not translate into temporal patterns. Instead, the use of boosters is relatively stable over the citation histories of the DSM.

The different patterns for these interactional markers point to the complexity of human language in scientific writings. One example of such complexity is that citations to the DSM are not consistently given in an instrumental context. This may be applicable to other research instruments, given the diversity of citation practices for material objects (Li et al., 2019). To address this complexity, these linguistic attributes can be employed as useful instruments for future works to automatically identify research instruments from scientific citations and texts, which will further help to construct a material-oriented history of science.

This study also evaluated the distribution of main verbs in all citances of the DSM. Two conclusions are drawn from the analysis. First, the verb profile of the DSM (even when based on all Method section citances) is very different from earlier findings based on all citations in the Method section (Bertin et al., 2016). This finding indicates that even under the citation contexts of the Method section, there are different and highly individual connections between individual citations (or instruments) and verbs (or actions taken in the research). This is consistent with how the Method section is conceptualized in the Create a Research Space (CARS) model, where the section serves to describe a localized research setting (Swales, 1990).

This individualized connection between verbs and citations raises questions about the extent to which we can use a general verb ranking or one scheme of citation context to analyze the Method section, given the vast diversity of research actions taken by researchers. Before fully transforming verbs into a valid research instrument, we need to build better knowledge about the categorization of research actions expressed by action verbs, with a comprehensive consideration of local factors, such as disciplinarity and the life cycle of the research objects represented by the citation. This research direction will be an important complement to existing works on full-text scientific publications.

Despite the individual nature of the connections between verbs and citations, verbs have proven to be a solid instrument to evaluate citation contexts at a somewhat aggregated level. By analyzing the similarity of verb profiles, we draw the same conclusion that older DSM versions are more strongly connected to the Method section than are newer ones.

5. CONCLUSION

The research reported in this paper offers a citation context analysis of how the DSM is cited in full-text psychological publications. Our results show that the re-instrumentality, or the extent to which the DSM is used in the Method section, varies as specific versions of the DSM go through their respective life stages. Over the first few years after its publication, a DSM version is increasingly cited in the Method section, which indicates that it is increasingly cited as a research instrument over time. This ratio does not seem to be influenced by the publication of the next version and only reaches its peak after around 10–15 years. Moreover, this changing level of instrumentality is accompanied by shifts in the use of some other important citation contexts within citation sentences. We found significantly different uses of hedges, attention markers, and some verbs as the DSM becomes more mature.

Our study offers an important quantitative examination of how citations serve as research instruments, one citation function that has not been extensively studied in quantitative science studies. Specifically, this research builds connections between quantitative evidence and theories of instrumentalization developed in qualitative studies in the field of science and technology studies. We found another aspect of how research instruments are constructed that is not fully addressed in the qualitative literature: After “physical” packaging, research instruments need to be re-instrumentalized in the space of scientific texts, as reflected in those texts and their citations.

This paper, despite the significance of its findings, represents only the first step towards a deeper understanding of how research instruments are represented in scientific texts. After all, it is only a case study of a unique research instrument on many different levels. To address this limitation, as the next step of this research project, we plan to conduct larger-scale quantitative analyses to examine the findings from the present study more comprehensively, especially the manner in which these research instruments are used differently across knowledge domains. Moreover, it is also possible that DSM is mentioned, instead of cited, in these publications, as DSM versions are “obliterated” into mere mentions over their citation histories (McCain, 2014). While this question cannot be investigated by this current study, it is an important question to be pursued in future works.

ACKNOWLEDGMENTS

Part of this work was done while the author was a visiting researcher at the Centre for Science and Technology Studies (CWTS) at Leiden University. The author thanks CWTS for providing access to its data infrastructure. The author would also like to thank Drs. Erjia Yan, Cassidy Sugimoto, Chaomei Chen, Jake Williams, Jane Greenberg, and Ali Kenner for their helps in the composition of this article. My thanks also go to Dr. Ludo Waltman and peer reviewers for their comments on the manuscript.

FUNDING INFORMATION

No funding was received for this research.

COMPETING INTERESTS

The author has no competing interests.

DATA AVAILABILITY

The data cannot be made available in a data repository because the provider of the data (Elsevier) does not allow this.

Notes

5

Many, if not most, of these method papers are representations of research instruments, especially when they are cited in the Method section, even though “research instrument” is not the framework adopted by Dr. Henry Small.

REFERENCES

American Psychiatric Association
. (
2013
).
Diagnostic and statistical manual of mental disorders
(5th ed.).
Washington, DC
:
American Psychiatric Publishing
.
Bertin
,
M.
, &
Atanassova
,
I.
(
2014
).
A study of lexical distribution in citation contexts through the IMRaD standard
.
PLOS Neglected Tropical Diseases
,
1
(
200,920
),
83
402
.
Bertin
,
M.
,
Atanassova
,
I.
,
Sugimoto
,
C. R.
, &
Lariviere
,
V.
(
2016
).
The linguistic patterns and rhetorical structure of citation context: an approach using n-grams
.
Scientometrics
,
109
(
3
),
1417
1434
.
Bloch
,
J.
(
2010
).
A concordance-based study of the use of reporting verbs as rhetorical devices in academic papers
.
Journal of Writing Research
,
2
(
2
),
219
244
.
Bowker
,
G. C.
, &
Star
,
S. L.
(
2000
).
Sorting things out: Classification and its consequences
.
Cambridge, MA
:
MIT Press
.
Chen
,
C.
,
Song
,
M.
, &
Heo
,
G. E.
(
2018
).
A scalable and adaptive method for finding semantically equivalent cue words of uncertainty
.
Journal of Informetrics
,
12
(
1
),
158
180
.
Chubin
,
D. E.
, &
Moitra
,
S. D.
(
1975
).
Content analysis of references: Adjunct or alternative to citation counting?
Social Studies of Science
,
5
(
4
),
423
441
.
Clarke
,
A. E.
, &
Fujimura
,
J. H.
(
1992
).
What tools? Which jobs? Why right?
In
A. E.
Clarke
&
J. H.
Fujimura
(Eds.)
The right tools for the job: At work in twentieth-century life sciences
(pp.
3
44
).
Princeton, NJ
:
Princeton University Press
.
Daston
,
L.
, &
Galison
,
P.
(
2010
).
Objectivity
.
New York, NY
:
Zone Books
.
Didegah
,
F.
,
Bowman
,
T. D.
, &
Holmberg
,
K.
(
2018
).
On the differences between citations and altmetrics: An investigation of factors driving altmetrics versus citations for Finnish articles
.
Journal of the Association for Information Science and Technology
,
69
(
6
),
832
843
.
Ding
,
Y.
,
Liu
,
X.
,
Guo
,
C.
, &
Cronin
,
B.
(
2013
).
The distribution of references across texts: Some implications for citation analysis
.
Journal of Informetrics
,
7
(
3
),
583
592
.
Engeström
,
Y.
(
1990
).
When is a tool? Multiple meanings of artifacts in human activity
. In
Y.
Engeström
(Ed.),
Learning, Working and Imagining: Twelve Studies in Activity Theory
(pp.
171
195
).
Helsinki
:
Orienta-Konsultit Oy
.
Flesch
,
R.
(
1948
).
A new readability yardstick
.
Journal of Applied Psychology
,
32
(
3
),
221
.
Fraassen
,
B. C. van
. (
2008
).
Scientific Representation: Paradoxes of Perspective
.
Oxford
:
Oxford University Press
.
Garfield
,
E.
(
1980
).
Citation measures of the influence of Robert K. Merton
.
Transactions of the New York Academy of Sciences
,
39
(
1 Series II
),
61
74
.
Hartley
,
J.
(
2016
).
Is time up for the Flesch measure of reading ease?
Scientometrics
,
107
(
3
),
1523
1526
.
Hartley
,
J.
,
Pennebaker
,
J.
, &
Fox
,
C.
(
2003
).
Abstracts, introductions and discussions: How far do they differ in style?
Scientometrics
,
57
(
3
),
389
398
.
Hayden
,
J. D.
(
2008
).
Readability in the British Journal of Surgery
.
British Journal of Surgery
,
95
(
1
),
119
124
.
Honnibal
,
M.
, &
Montani
,
I.
(
2017
).
spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing
.
To Appear
.
Hooten
,
P. A.
(
1991
).
Frequency and functional use of cited documents in information science
.
Journal of the American Society for Information Science
,
42
(
6
),
397
404
.
Howison
,
J.
, &
Bullard
,
J.
(
2015
).
Software in the scientific literature: Problems with seeing, finding, and using software mentioned in the biology literature
.
Journal of the Association for Information Science and Technology
,
67
(
9
),
2137
2155
.
Huang
,
D.
(
2014
).
Genre analysis of moves in medical research articles
.
Stylus
,
5
(
1
),
7
17
.
Hyland
,
K.
(
1999
).
Academic attribution: Citation and the construction of disciplinary knowledge
.
Applied Linguistics
,
20
(
3
),
341
367
.
Hyland
,
K.
(
2003
).
Self-citation and self-reference: Credibility and promotion in academic publication
.
Journal of the American Society for Information Science and Technology
,
54
(
3
),
251
259
.
Hyland
,
K.
(
2005a
).
Metadiscourse: Exploring interaction in writing
.
London
:
A&C Black
.
Hyland
,
K.
(
2005b
).
Stance and engagement: a model of interaction in academic discourse
.
Discourse Studies
,
7
(
2
),
173
192
.
Jasanoff
,
S.
(
2004
).
States of knowledge: the co-production of science and the social order
.
London
:
Routledge
.
Jha
,
R.
,
Jbara
,
A.-A.
,
Qazvinian
,
V.
, &
Radev
,
D. R.
(
2017
).
NLP-driven citation analysis for scientometrics
.
Natural Language Engineering
,
23
(
1
),
93
130
.
Kanoksilapatham
,
B.
(
2012
).
Facilitating scholarly publication: Genre characteristics of English research article Introductions and Methods
.
3L: Language, Linguistics, Literature
,
18
(
4
).
Knorr-Cetina
,
K.
(
1999
).
Epistemic cultures: How the sciences make knowledge
.
Cambridge, MA
:
Harvard University Press
.
Lamers
,
W.
,
van Eck
,
N. J.
,
Waltman
,
L.
, &
Hoos
,
H.
(
2018
).
Patterns in citation context: The case of the field of scientometrics
.
23rd International Conference on Science and Technology Indicators (STI 2018)
,
September 12–14
,
Leiden, The Netherlands
.
Lamurias
,
A.
, &
Couto
,
F.
(
2019
).
Text mining for bioinformatics using biomedical literature
.
Encyclopedia of Bioinformatics and Computational Biology
,
1
,
602
611
.
Larivière
,
V.
,
Sugimoto
,
C. R.
,
Macaluso
,
B.
,
Milojević
,
S.
,
Cronin
,
B.
, &
Thelwall
,
M.
(
2014
).
arXiv E-prints and the journal of record: An analysis of roles and relationships
.
Journal of the Association for Information Science and Technology
,
65
(
6
),
1157
1169
.
Latour
,
B.
(
1987
).
Science in action: How to follow scientists and engineers through society
.
Cambridge, MA
:
Harvard University Press
.
Latour
,
B.
, &
Woolgar
,
S.
(
1979
).
Laboratory life: The construction of scientific facts
.
Princeton, NJ
:
Princeton University Press
.
Leydesdorff
,
L.
,
Ràfols
,
I.
, &
Milojevic
,
S.
(
2020
).
Bridging the divide between qualitative and quantitative science studies
.
Cambridge, MA
:
MIT Press
.
Li
,
K.
,
Chen
,
P.-Y.
, &
Yan
,
E.
(
2019
).
Challenges of measuring software impact through citations: An examination of the lme4 R package
.
Journal of Informetrics
,
13
(
1
),
449
461
.
Li
,
K.
,
Yan
,
E.
, &
Feng
,
Y.
(
2017
).
How is R cited in research outputs? Structure, impacts, and citation standard
.
Journal of Informetrics
,
11
(
4
),
989
1002
.
McCain
,
K. W.
(
2014
).
Obliteration by incorporation
. In
B.
Cronin
&
C. R.
Sugimoto
(Eds.),
Beyond bibliometrics: Harnessing multidimensional indicators of scholarly impact
(pp.
129
149
).
Cambridge, MA
:
MIT Press
.
McCain
,
K. W.
, &
Salvucci
,
L. J.
(
2006
).
How influential is Brooks’ law? A longitudinal citation context analysis of Frederick Brooks’ The Mythical Man-Month
.
Journal of Information Science
,
32
(
3
),
277
295
.
McCain
,
K. W.
, &
Turner
,
K.
(
1989
).
Citation context analysis and aging patterns of journal articles in molecular genetics
.
Scientometrics
,
17
(
1
),
127
163
.
Millon
,
T. E.
, &
Klerman
,
G. L.
(
1986
).
Contemporary directions in psychopathology: Toward the DSM-IV
.
New York, NY
:
Guilford Press
.
Nakhli
,
J.
,
El Kissi
,
Y.
,
Bouhlel
,
S.
,
Amamou
,
B.
,
Nabli
,
T. A.
,
Nasr
,
S. Ben
, &
Ali
,
B. B. H.
(
2014
).
Reliability and validity of the Arizona Sexual Experiences Scale-Arabic version in Tunisian patients with schizophrenia
.
Comprehensive Psychiatry
,
55
(
6
),
1473
1477
.
Oleinik
,
A.
,
Kirdina-Chandler
,
S.
,
Popova
,
I.
, &
Shatalova
,
T.
(
2017
).
On academic reading: citation patterns and beyond
.
Scientometrics
,
113
(
1
),
417
435
.
Pröll
,
S.
, &
Rauber
,
A.
(
2013
).
Scalable data citation in dynamic, large databases: Model and reference implementation
.
Proceedings of the 2013 IEEE International Conference on Big Data (Big Data 2013)
(pp.
307
312
).
Regier
,
D. A.
,
Narrow
,
W. E.
,
Kuhl
,
E. A.
, &
Kupfer
,
D. J.
(
2009
).
The conceptual development of DSM-V
.
American Journal of Psychiatry
,
166
(
6
),
645
650
.
Rheinberger
,
H.-J.
(
1997
).
Toward a history of epistemic things: Synthesizing proteins in the test tube
.
Redwood City, CA
:
Stanford University Press
.
Roth
,
C.
, &
Basov
,
N.
(
2020
).
The socio-semantic space of John Mohr
.
Poetics
,
101437
. .
Small
,
H.
(
1982
).
Citation context analysis
.
Progress in Communication Sciences
,
3
,
287
310
.
Small
,
H.
(
2018
).
Characterizing highly cited method and non-method papers using citation contexts: The role of uncertainty
.
Journal of Informetrics
,
12
(
2
),
461
480
.
Small
,
H.
,
Boyack
,
K. W.
, &
Klavans
,
R.
(
2019
).
Citations and certainty: A new interpretation of citation counts
.
Scientometrics
,
118
(
3
),
1079
1092
.
Small
,
H.
,
Tseng
,
H.
, &
Patek
,
M.
(
2017
).
Discovering discoveries: Identifying biomedical discoveries using citation contexts
.
Journal of Informetrics
,
11
(
1
),
46
62
.
Smith
,
A. M.
,
Katz
,
D. S.
, &
Niemeyer
,
K. E.
(
2016
).
Software citation principles
.
PeerJ Computer Science
,
2
,
e86
.
Swales
,
J.
(
1990
).
Genre analysis: English in academic and research settings
.
Cambridge
:
Cambridge University Press
.
Tagay
,
S.
,
Herpertz
,
S.
,
Langkafel
,
M.
, &
Senf
,
W.
(
2005
).
Posttraumatic stress disorder in a psychosomatic outpatient clinic: Gender effects, psychosocial functioning, sense of coherence, and service utilization
.
Journal of Psychosomatic Research
,
58
(
5
),
439
446
.
Tang
,
R.
, &
Safer
,
M. A.
(
2008
).
Author-rated importance of cited references in biology and psychology publications
.
Journal of Documentation
,
64
(
2
),
246
272
.
Thelwall
,
M.
(
2019
).
The rhetorical structure of science? A multidisciplinary analysis of article headings
.
Journal of Informetrics
,
13
(
2
),
555
563
.
Torres
,
A. R.
,
Ferrão
,
Y. A.
,
Shavitt
,
R. G.
,
Diniz
,
J. B.
,
Costa
,
D. L. C.
,
Do Rosário
,
M. C.
,
Miguel
,
E. C.
, &
Fontenelle
,
L. F.
(
2014
).
Panic disorder and agoraphobia in OCD patients: Clinical profile and possible treatment implications
.
Comprehensive Psychiatry
,
55
(
3
),
588
597
.
van Strien
,
T.
,
van der Zwaluw
,
C. S.
, &
Engels
,
R. C. M. E.
(
2010
).
Emotional eating in adolescents: A gene (SLC6A4/5-HTT)–Depressive feelings interaction analysis
.
Journal of Psychiatric Research
,
44
(
15
),
1035
1042
.
Van Wesel
,
M.
,
Wyatt
,
S.
, &
ten Haaf
,
J.
(
2014
).
What a difference a colon makes: How superficial factors influence subsequent citation
.
Scientometrics
,
98
(
3
),
1601
1615
.
Voos
,
H.
, &
Dagaev
,
K. S.
(
1976
).
Are all citations equal? Or, did we op. cit. your idem?
Journal of Academic Librarianship
,
1
(
6
),
19
21
.
Watson
,
P. J.
,
Ross
,
D. F.
, &
Morris
,
R. J.
(
2003
).
Borderline personality traits correlate with death penalty decisions
.
Personality and Individual Differences
,
35
(
2
),
421
429
.
Young
,
A.
(
1997
).
The harmony of illusions: Inventing post-traumatic stress disorder
.
Princeton, NJ
:
Princeton University Press
.
Zhao
,
D.
,
Cappello
,
A.
, &
Johnston
,
L.
(
2017
).
Functions of uni-and multi-citations: Implications for weighted citation analysis
.
Journal of Data and Information Science
,
2
(
1
),
51
69
.
Zhao
,
D.
, &
Strotmann
,
A.
(
2020
).
Deep and narrow impact: Introducing location filtered citation counting
.
Scientometrics
,
122
(
1
),
503
517
.
Zhao
,
M.
,
Yan
,
E.
, &
Li
,
K.
(
2018
).
Data set mentions and citations: A content analysis of full-text publications
.
Journal of the Association for Information Science and Technology
,
69
(
1
),
32
46
.

APPENDIX A: SELECTED METHOD-SECTION TITLES

TitleCount in paper sample
methods 40,735 
method 39,616 
materials and methods 11,901 
methodology 2,981 
material and methods 2,699 
experimental procedures 1,380 
data and methods 1,237 
methods and materials 953 
subjects and methods 569 
experimental design 433 
experimental results 387 
patients and methods 375 
research methodology 312 
research method 286 
materials and method 258 
research methods 230 
data and method 214 
data and methodology 173 
general method 134 
general methods 123 
participants and methods 112 
material and method 112 
TitleCount in paper sample
methods 40,735 
method 39,616 
materials and methods 11,901 
methodology 2,981 
material and methods 2,699 
experimental procedures 1,380 
data and methods 1,237 
methods and materials 953 
subjects and methods 569 
experimental design 433 
experimental results 387 
patients and methods 375 
research methodology 312 
research method 286 
materials and method 258 
research methods 230 
data and method 214 
data and methodology 173 
general method 134 
general methods 123 
participants and methods 112 
material and method 112 

APPENDIX B: EXAMPLES OF HOW HYLAND’S METADISCOURSE EXPRESSIONS ARE USED IN OUR PAPER SAMPLE

CategoryDSM-citation sentence with the target phrase highlightedCitation
Hedges Perhaps with these cautionary notes in mind, the American Psychiatric Association has thus far taken a conservative approach to proposing specific new diagnoses for electronic media in the forthcoming DSM-V (American Psychiatric Association, 2010).” (van Strien, van der Zwaluw et al., 2010
Boosters “Our differentiated evaluation of the prevalence of trauma or PTSD demonstrates that not every trauma on the PDS trauma checklist equates to a psychic trauma according to DSM-IV [1].” (Tagay, Herpertz et al., 2005
Attitude markers Unfortunately, however, we were unable to investigate how the recently proposed DSM-5 subtypes of hypochondria (i.e. care seeking vs. care avoidant types) [63] relate to OCD with PD/AG vs. OCD without PD/AG.” (Torres, Ferrão et al., 2014
Self-mention We have performed this study with one hundred first outpatients with a diagnosis of schizophrenia (DSM-IV TR criteria) [12] from the outpatient psychiatric service of Farhat Hached Hospital in Sousse.” (Nakhli, El Kissi et al., 2014
Engagement markers “Such an analysis obviously receives some confirmation in observations that borderline personalities display an unstable identity (American Psychiatric Association, 1994) and that borderline traits correlated inversely with death penalty support in the present project.” (Watson, Ross, & Morris, 2003
CategoryDSM-citation sentence with the target phrase highlightedCitation
Hedges Perhaps with these cautionary notes in mind, the American Psychiatric Association has thus far taken a conservative approach to proposing specific new diagnoses for electronic media in the forthcoming DSM-V (American Psychiatric Association, 2010).” (van Strien, van der Zwaluw et al., 2010
Boosters “Our differentiated evaluation of the prevalence of trauma or PTSD demonstrates that not every trauma on the PDS trauma checklist equates to a psychic trauma according to DSM-IV [1].” (Tagay, Herpertz et al., 2005
Attitude markers Unfortunately, however, we were unable to investigate how the recently proposed DSM-5 subtypes of hypochondria (i.e. care seeking vs. care avoidant types) [63] relate to OCD with PD/AG vs. OCD without PD/AG.” (Torres, Ferrão et al., 2014
Self-mention We have performed this study with one hundred first outpatients with a diagnosis of schizophrenia (DSM-IV TR criteria) [12] from the outpatient psychiatric service of Farhat Hached Hospital in Sousse.” (Nakhli, El Kissi et al., 2014
Engagement markers “Such an analysis obviously receives some confirmation in observations that borderline personalities display an unstable identity (American Psychiatric Association, 1994) and that borderline traits correlated inversely with death penalty support in the present project.” (Watson, Ross, & Morris, 2003

Author notes

Handling Editor: Ludo Waltman

This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. For a full description of the license, please visit https://creativecommons.org/licenses/by/4.0/legalcode.