Abstract

The study of signed languages allows the dissociation of sensorimotor and cognitive neural components of the language signal. Here we investigated the neurocognitive processes underlying the monitoring of two phonological parameters of sign languages: handshape and location. Our goal was to determine if brain regions processing sensorimotor characteristics of different phonological parameters of sign languages were also involved in phonological processing, with their activity being modulated by the linguistic content of manual actions. We conducted an fMRI experiment using manual actions varying in phonological structure and semantics: (1) signs of a familiar sign language (British Sign Language), (2) signs of an unfamiliar sign language (Swedish Sign Language), and (3) invented nonsigns that violate the phonological rules of British Sign Language and Swedish Sign Language or consist of nonoccurring combinations of phonological parameters. Three groups of participants were tested: deaf native signers, deaf nonsigners, and hearing nonsigners. Results show that the linguistic processing of different phonological parameters of sign language is independent of the sensorimotor characteristics of the language signal. Handshape and location were processed by different perceptual and task-related brain networks but recruited the same language areas. The semantic content of the stimuli did not influence this process, but phonological structure did, with nonsigns being associated with longer RTs and stronger activations in an action observation network in all participants and in the supramarginal gyrus exclusively in deaf signers. These results suggest higher processing demands for stimuli that contravene the phonological rules of a signed language, independently of previous knowledge of signed languages. We suggest that the phonological characteristics of a language may arise as a consequence of more efficient neural processing for its perception and production.

INTRODUCTION

Valuable insights into the neuroanatomy of language and cognition can be gained from the study of signed languages. Signed languages differ dramatically from spoken languages with respect both to the articulators (the hands vs. the vocal tract) and to the perceptual system supporting comprehension (vision vs. audition). However, linguistically (Sutton-Spence & Woll, 1999), cognitively (Rudner, Andin, & Rönnberg, 2009), and neurobiologically (Corina, Lawyer, & Cates, 2012; MacSweeney, Capek, Campbell, & Woll, 2008; Söderfeldt, Rönnberg, & Risberg, 1994), there are striking similarities. Thus, studying signed languages allows sensorimotor mechanisms to be dissociated from cognitive mechanisms, both behaviorally and neurobiologically.

In this study, we investigated the neural networks underlying monitoring of the handshape and location (two phonological components of sign languages) of manual actions that varied in phonological structure and semantic content. Our main goal was to determine if brain regions involved in processing sensorimotor characteristics of the language signal were also involved in phonological processing, with their activity being modulated by the linguistic content of manual actions.

The semantic purpose of language—the sharing of meaning—is similar across signed and spoken languages. However, the phonological level of language processing may be specifically related to the sensorimotor characteristics of the language signal. Spoken language phonology relates to sound patterning in the sublexical structure of words. Sign language phonology relates to the sublexical structure of signs and in particular the patterning of handshape, hand location in relation to the body, and hand movement (Emmorey, 2002). Phonology is generally considered to be arbitrarily related to semantics. In signed languages, however, phonology is not always independent of meaning (for an overview, see Gutiérrez, Williams, Grosvald, & Corina, 2012), and this relation seems to influence language processing (Grosvald, Lachaud, & Corina, 2012; Thompson, Vinson, & Vigliocco, 2010) and its neural underpinning (Rudner, Karlsson, Gunnarsson, & Rönnberg, 2013; Gutiérrez, Müller, Baus, & Carreiras, 2012).

Speech-based phonological processing skill relies on mechanisms whose neural substrate is located in the posterior portion of the left inferior frontal gyrus (IFG) and the ventral premotor cortex (see Price, 2012, for a review). The posterior parts of the junction of the parietal and temporal lobes bilaterally (Hickok & Poeppel, 2007), particularly the left and right supramarginal gyri (SMG), are also involved in speech-based phonology, activating when participants make decisions about the sounds of words (i.e., their phonology) in contrast to decisions about their meanings (i.e., their semantics; Hartwigsen et al., 2010; Devlin, Matthews, & Rushworth, 2003; McDermott, Petersen, Watson, & Ojemann, 2003; Price, Moore, Humphreys, & Wise, 1997).

The phonology of sign language is processed by left-lateralized neural networks similar to those that support speech phonology (MacSweeney, Waters, Brammer, Woll, & Goswami, 2008; Emmorey, Mehta, & Grabowski, 2007), although activations in the left IFG are more anterior for sign language (Rudner et al., 2013; MacSweeney, Brammer, Waters, & Goswami, 2009; MacSweeney, Waters, et al., 2008). Despite these similarities, it is not clear to what extent the processing of the specific phonological parameters of sign languages, such as handshape, location, and movement, recruits functionally different neural networks. Investigation of the mechanisms of sign phonology have often focused separately on sign handshape (Andin, Rönnberg, & Rudner, 2014; Andin et al., 2013; Grosvald et al., 2012; Wilson & Emmorey, 1997) and sign location (Colin, Zuinen, Bayard, & Leybaert, 2013; MacSweeney, Waters, et al., 2008). Studies that have compared these two phonological parameters identified differences in comprehension and production psycholinguistically (e.g., Orfanidou, Adam, McQueen, & Morgan, 2009; Carreiras, Gutiérrez-Sigut, Baquero, & Corina, 2008; Dye & Shih, 2006; Emmorey, McCullough, & Brentari, 2003), developmentally (e.g., Morgan, Barrett-Jones, & Stoneham, 2007; Karnopp, 2002; Siedlecki & Bonvillian, 1993), and neuropsychologically (Corina, 2000). In particular, the neural signature of handshape and location-based primes has been found to differ between signs and nonsigns and further interact with the semantic properties of signs (Grosvald et al., 2012; Gutiérrez, Müller, et al., 2012). However, no study to date has investigated the differences in neural networks underlying monitoring of handshape and location.

Handshape and location can be conceptualized differently in terms of their perceptual and linguistic properties. In linguistic (phonological) terms, location refers to the position of the signing hand in relation to the body. The initial location has been referred to as the equivalent of syllable onset in spoken languages (Brentari, 2002), with electrophysiological evidence suggesting that location triggers the activation of lexical candidates in signed languages, indicating a function similar to that of the onset in spoken word recognition (Gutiérrez, Müller, et al., 2012; Gutiérrez, Williams, et al., 2012). Perceptually, monitoring of location relates to the tracking of visual objects in space and in relation to equivalent positions relative to the viewer's body. As such, it is expected that extraction of the feature of location will recruit dorsal visual areas, which are involved in visuospatial processing and visuomotor transformations (Ungerleider & Haxby, 1994; Milner & Goodale, 1993), and resolve spatial location of objects. Parietal areas involved in the identification of others' body parts (Felician et al., 2009) and those involved in self-reference, such as medial prefrontal, anterior cingulate, and precuneus, could also be involved in the extraction of this feature (Northoff & Bermpohl, 2004).

Handshape refers to contrastive configurations of the fingers (Sandler & Lillo-Martin, 2006). It has been shown that deaf signers are faster and more accurate than hearing nonsigners at identifying handshape during a monitoring task and that lexicalized signs are more easily identified than nonlexicalized signs (Grosvald et al., 2012). In terms of lexical retrieval, handshape seems to play a greater role in later stages than location (Gutiérrez, Müller, et al., 2012), possibly by constraining the set of activated lexical items. From a perceptual point of view, monitoring of handshape is likely to recruit ventral visual and parietal areas involved in the processing of object categories and forms—in particular regions that respond more to hand stimuli than to other body parts or objects, such as the left lateral occipitotemporal cortex, the extrastriate body area, the fusiform body area, the superior parietal lobule, and the intraparietal sulcus (Bracci, Ietswaart, Peelen, & Cavina-Pratesi, 2010; Op de Beeck, Brants, Baeck, & Wagemans, 2010; Vingerhoets, de Lange, Vandemaele, Deblaere, & Achten, 2002; Jordan, Heinze, Lutz, Kanowski, & Jancke, 2001; Alivesatos & Petrides, 1997; Ungerleider & Haxby, 1994; Milner & Goodale, 1993). Motor areas processing specific muscle–skeletal configurations are also likely to be recruited (Hamilton & Grafton, 2009; Gentilucci & Dalla Volta, 2008). Thus, it is likely that different networks will be recruited for the perceptual and motoric processing of these phonological components. Evidence showing that phonological priming of location and handshape modulates components of the ERP signal differently for signs and nonsigns and for native and non-native signers suggests that these networks may be modulated by the semantic content of the signs as well as the sign language experience of the participants (Gutiérrez, Müller, et al., 2012).

In this study, we used a sign language equivalent of a phoneme-monitoring task (Grosvald et al., 2012) to investigate the neural networks underlying processing of two phonological components (handshape and location). Participants were instructed to press a button when they saw a sign that was produced in a cued location or that contained a cued handshape. Although our monitoring task taps into processes underlying sign language comprehension, it can be performed by both signers and nonsigners. Our stimuli varied in phonological structure and semantic content and included (1) signs of a familiar sign language (British Sign Language, BSL), which deliver semantic and phonological information; (2) signs of an unfamiliar sign language (Swedish Sign Language, SSL), chosen to be phonologically possible but nonlexicalized for BSL signers, delivering mainly phonological information, and thus equivalent to pseudosigns; and (3) invented nonsigns, which violate the phonological rules of BSL and SSL or contain nonoccurring combinations of phonological parameters in order to minimize the amount of phonological information that can be extracted from the stimuli. By testing different groups of participants (deaf native signers, deaf nonsigners, and hearing nonsigners), we were able to dissociate the influence of hearing status and sign language experience. This design allows us to contrast extraction of handshape and location in a range of linguistic contexts, with and without sign language knowledge and with and without auditory deprivation. Thus, it enables us to determine whether neural networks are sensitive to the phonological structure of natural language even when that structure has no linguistic significance. This cannot easily be achieved merely by studying language in the spoken domain, as all hearing individuals with typical development use a speech-based language sharing at least some phonological structure with other spoken languages.

We hypothesize that different perceptual and motor brain regions will be recruited for the processing of handshape and location, and this will be observed in all groups of participants, independently of their hearing status and sign language knowledge. Regarding visual processing networks, we expect dorsal visual areas to be more active during the monitoring of location and ventral visual areas to be more active while monitoring handshape (effect of task). If visual processing mechanisms are recruited for phonological processing, different patterns of activation will be found for deaf signers (compared to nonsigners) in ventral and dorsal visual areas for the handshape and location task (respectively). On the other hand, if phonological processing is independent of the sensorimotor characteristics of the language signal, the handshape and location tasks will not recruit ventral and dorsal visual areas differently in signers and nonsigners (Group × Task interaction). We also hypothesize that the semantic and phonological structure of signs will modulate neurocognitive mechanisms underpinning phoneme monitoring, with effects seen behaviorally and in the neuroimaging data. Specifically, we expect meaningful signs to differentially recruit regions from a large-scale semantic network including the posterior inferior parietal cortex, STS, parahippocampal cortex, posterior cingulate, and pFC (including IFG; Binder, Desai, Graves, & Conant, 2009). We also hypothesize that stimuli varying in phonological structure will differentially recruit regions involved in phonological processing, such as the left IFG, the ventral premotor cortex, and the posterior parts of the junction of the parietal and temporal lobes, including the SMG (Group × Stimulus type interaction).

METHODS

This study is part of a larger study involving cross-linguistic comparisons and assessments of cross-modal plasticity in signers and nonsigners. Some results of this larger study have already been published (Cardin et al., 2013), and others will be published elsewhere.

Participants

There were three groups of participants:

  • (A) 

    Deaf signers: Congenitally severely-to-profoundly deaf individuals who have deaf parents and are native signers of BSL: n = 15; age = 38.37 ± 3.22 years; sex = 6 male, 9 female; better-ear pure tone average (1 kHz, 2 kHz, 4 kHz; maximum output of equipment = 100 dB) = 98.2 ± 2.4 dB; non-verbal IQ, as measured with the blocks design subtest of the Wechsler Abbreviated Scale of Intelligence (WASI) = 62.67 ± 1.5. Participants in this group were not familiar with SSL.

  • (B) 

    Deaf nonsigners: Congenitally or early (before 3 years) severely-to-profoundly deaf individuals with hearing parents, who are native speakers of English accessing language through speechreading, and who have never learned a sign language: n = 10; age = 49.8 ± 1.7 years; sex = 6 male, 4 female; pure tone average = 95.2 ± 2.6 dB; WASI = 64.8 ± 1.8.

  • (C) 

    Hearing nonsigners: Participants with normal hearing who are native speakers of English with no knowledge of a sign language: n = 18; age = 37.55 ± 2.3 years; sex = 9 male, 9 female. WASI = 60.93 ± 2.1.

Participants in the deaf signers and hearing nonsigners groups were recruited from local databases. Most of the participants in the deaf nonsigners group were recruited through an association of former students of a local oral education school for deaf children. Sign language knowledge was an exclusion criterion for the deaf nonsigners and hearing nonsigner groups. Because of changing attitudes toward sign language, deaf people are now more likely to be interested in learning to sign as young adults, even if they were raised in a completely oral environment and developed a spoken language successfully. For this reason, all the participants in the deaf nonsigners were more than 40 years. The average age of this group was significantly different from that of the deaf signers (p = .019) and the hearing nonsigners (p = .0012). The number of male and female participants was also different across groups. For this reason, age and sex were entered as covariates in all our analyses. No other parameter was significantly different across groups.

All participants gave their written informed consent. This study was approved by the UCL ethics committee. All participants traveled to Birkbeck-UCL Centre of Neuroimaging in London to take part in the study and were paid a small fee for their time and compensated for their travel and accommodation expenses.

Stimuli

Our experiment was designed with four types of stimuli (Tables 1 and 2): BSL-only signs (i.e., not lexicalized in SSL), SSL-only signs (i.e., not lexicalized in BSL), cognates (i.e., signs with identical form and meaning in BSL and SSL), and nonsigns (i.e., sign-like items that are neither signs of BSL nor SSL and made by specifically violating phonotactic rules or including highly unusual or nonoccurring combinations of phonological parameters).

Table 1. 

Stimuli—BSL, Cognates, and SSL

BSLCognatesSSL
SignTypePartsSignTypePartsSignEnglish NameTypeParts
afternoon 1L alarm 2AS äcklig disgusting 1L 
amazed 2S announce 2S afton evening 1L 
argue 2S Belgium 1L ambitiös ambitious 2S 
bedroom 1L belt 2S anka duck 2S 
believe 1L/2AS bicycle 2S anställd employee 2S 
biscuit 1L bomb 2S april April 1L 
can't-be-bothered 1L can't-believe 1L/2AS avundssjuk envious 1L 
castle 2S cards 2AS bakelse fancy pastry 2AS 
cheese 2AS clock 2AS bättre better 1L 
cherry 1L clothes-peg 2AS bedrägeri fraud 1L 
chocolate 1L digital 2S/2S beröm praise 1L/2AS 
church 2S dive 2S bevara keep 2S 
cook 2S dream 1L billig cheap 10 
copy 2AS Europe 10 blyg shy 1L 
cruel 1L gossip 10 böter fine 2AS 
decide 1L/2AS hearing-aid 1L brk trouble 2S 
dog 10 Holland 2S broms brake 2S 
drill 2AS Japan 2S cognac brandy 10 
DVD 2AS letter 2AS ekorre squirrel 1L 
easy 1L light-bulb 1L farfar grandfather 1L 
evening 1L meet 2S filt rug 2AS 
February 2S/2S monkey 2S final final 2AS 
finally 2S new 2AS historia history 10 
finish 2S Norway 10 Indien India 1L 
fire 2S paint 2S kakao cocoa 1L/10 
flower 1L Paris 2S kalkon turkey (bird) 1L 
give-it-a-try 1L perfume 1L kalsonger underpants 1L 
helicopter 2AS pool 2AS korv sausage 2AS 
horrible 1L protect 2AS kväll evening 2AS 
house 2S Scotland 1L lördag Saturday 10 
ice-skate 2S shampoo 2S modig brave 2S 
live 1L sick 1L modig brave 1L 
luck 1L sign-language 2S partner partner 2S 
navy 2S ski 2S pommes frites French fries 2S 
silver 2S slap 10 rektor headmaster 1L 
sing 2S smile 1L rövare robber 2AS 
soldier 1L stir 2AS sambo cohabitant 1L/2AS 
strawberry 1L stomach-ache 2S service service 2AS 
strict 1L summarise 2S soldat soldier 2S 
theatre 2AS swallow 1L strut cone 2AS 
Thursday 2AS Switzerland 1L svamp mushroom 2AS 
toilet 1L tie 2AS sylt jam 1L 
tree 2AS tomato 2AS tända ignite 2AS 
trophy 2S translate 2AS välling gruel 1L 
wait 2S trousers 2S varmare hotter 1L 
Wales 10 violin 2AS verkstad workshop 10/2AS 
work 2AS weight 2S yngre younger 1L 
worried 2S yesterday 1L yoghurt yoghurt 1L 
BSLCognatesSSL
SignTypePartsSignTypePartsSignEnglish NameTypeParts
afternoon 1L alarm 2AS äcklig disgusting 1L 
amazed 2S announce 2S afton evening 1L 
argue 2S Belgium 1L ambitiös ambitious 2S 
bedroom 1L belt 2S anka duck 2S 
believe 1L/2AS bicycle 2S anställd employee 2S 
biscuit 1L bomb 2S april April 1L 
can't-be-bothered 1L can't-believe 1L/2AS avundssjuk envious 1L 
castle 2S cards 2AS bakelse fancy pastry 2AS 
cheese 2AS clock 2AS bättre better 1L 
cherry 1L clothes-peg 2AS bedrägeri fraud 1L 
chocolate 1L digital 2S/2S beröm praise 1L/2AS 
church 2S dive 2S bevara keep 2S 
cook 2S dream 1L billig cheap 10 
copy 2AS Europe 10 blyg shy 1L 
cruel 1L gossip 10 böter fine 2AS 
decide 1L/2AS hearing-aid 1L brk trouble 2S 
dog 10 Holland 2S broms brake 2S 
drill 2AS Japan 2S cognac brandy 10 
DVD 2AS letter 2AS ekorre squirrel 1L 
easy 1L light-bulb 1L farfar grandfather 1L 
evening 1L meet 2S filt rug 2AS 
February 2S/2S monkey 2S final final 2AS 
finally 2S new 2AS historia history 10 
finish 2S Norway 10 Indien India 1L 
fire 2S paint 2S kakao cocoa 1L/10 
flower 1L Paris 2S kalkon turkey (bird) 1L 
give-it-a-try 1L perfume 1L kalsonger underpants 1L 
helicopter 2AS pool 2AS korv sausage 2AS 
horrible 1L protect 2AS kväll evening 2AS 
house 2S Scotland 1L lördag Saturday 10 
ice-skate 2S shampoo 2S modig brave 2S 
live 1L sick 1L modig brave 1L 
luck 1L sign-language 2S partner partner 2S 
navy 2S ski 2S pommes frites French fries 2S 
silver 2S slap 10 rektor headmaster 1L 
sing 2S smile 1L rövare robber 2AS 
soldier 1L stir 2AS sambo cohabitant 1L/2AS 
strawberry 1L stomach-ache 2S service service 2AS 
strict 1L summarise 2S soldat soldier 2S 
theatre 2AS swallow 1L strut cone 2AS 
Thursday 2AS Switzerland 1L svamp mushroom 2AS 
toilet 1L tie 2AS sylt jam 1L 
tree 2AS tomato 2AS tända ignite 2AS 
trophy 2S translate 2AS välling gruel 1L 
wait 2S trousers 2S varmare hotter 1L 
Wales 10 violin 2AS verkstad workshop 10/2AS 
work 2AS weight 2S yngre younger 1L 
worried 2S yesterday 1L yoghurt yoghurt 1L 

The table lists the signs used in this study, including the number of component parts and the type of sign. BSL = BSL signs not lexicalized in SSL; Cognates = signs with identical form and meaning in BSL and SSL; SSL = SSL signs not lexicalized in BSL. Types of sign: 10 = one-handed sign not in contact with the body; 1L = one-handed sign in contact with the body (including the nondominant arm); 2S = symmetrical two-handed sign, both hands active and with the same handshape; 2AS = asymmetrical two-handed sign, one hand acts on the other hand; handshapes may be the same or different. Parts: 1 = 1-part/1 syllable; 2 = 2-part/2 syllables.

Table 2. 

Nonsigns

IDTypePartsOdd Feature(s)
2AS point of contact 
10 handshape change + orientation change 
1L handshape change + higher second location 
2AS location 
2S 2 different handshapes 
2AS point of contact 
2S orientation 
2AS location 
12 2S location 
13 2S handshape 
14 1L point of contact 
15 2AS handshape 
17 1L handshape, location + upward movement 
21 1L point of contact 
23 1L orientation change 
27 2S location change 
34 2AS point of contact + 2 different handshapes 
36 1L contralateral location on head 
37 2AS point of contact 
39 1L contralateral location on shoulder + orientation change 
41 1L location + handshape change 
43 1L location change 
44 2S orientation change + handshape change 
47 1L point of contact 
51 1L point of contact 
52 1L location + handshape change 
53 1L upward movement 
55 2S point of contact 
56 2S two different handshapes 
58 1L point of contact 
61 2S two different handshapes + point of contact 
62 10 movement 
64 2AS point of contact 
68 1L handshape change 
73 1L point of contact 
75 1L handshape 
79 1L point of contact 
81 1L point of contact 
83 1L handshape change 
85 1L movement 
89 2S location change + upward movement 
90 2S location change 
93 2S change to different handshapes 
96 2S location change 
98 1L 2 handshape changes 
99 1L handshape change + location change 
102 1L location change + upward movement 
103 1L location change + handshape change 
IDTypePartsOdd Feature(s)
2AS point of contact 
10 handshape change + orientation change 
1L handshape change + higher second location 
2AS location 
2S 2 different handshapes 
2AS point of contact 
2S orientation 
2AS location 
12 2S location 
13 2S handshape 
14 1L point of contact 
15 2AS handshape 
17 1L handshape, location + upward movement 
21 1L point of contact 
23 1L orientation change 
27 2S location change 
34 2AS point of contact + 2 different handshapes 
36 1L contralateral location on head 
37 2AS point of contact 
39 1L contralateral location on shoulder + orientation change 
41 1L location + handshape change 
43 1L location change 
44 2S orientation change + handshape change 
47 1L point of contact 
51 1L point of contact 
52 1L location + handshape change 
53 1L upward movement 
55 2S point of contact 
56 2S two different handshapes 
58 1L point of contact 
61 2S two different handshapes + point of contact 
62 10 movement 
64 2AS point of contact 
68 1L handshape change 
73 1L point of contact 
75 1L handshape 
79 1L point of contact 
81 1L point of contact 
83 1L handshape change 
85 1L movement 
89 2S location change + upward movement 
90 2S location change 
93 2S change to different handshapes 
96 2S location change 
98 1L 2 handshape changes 
99 1L handshape change + location change 
102 1L location change + upward movement 
103 1L location change + handshape change 

The table describes the composition of the nonsigns used in this study, including their component parts and type of sign. Nonsigns: sign-like items that are neither signs of BSL nor SSL and violate phonotactic rules of both languages. Types of sign: 10 = one-handed sign not in contact with the body; 1L = one-handed sign in contact with the body (including the non-dominant arm); 2S = symmetrical two-handed sign, both hands active and with the same handshape; 2AS = asymmetrical two-handed sign, one hand acts on the other hand; handshapes may be same or different. Parts: 1 = 1-part/1 syllable; 2 = 2-part/2 syllables.

Forty-eight video clips (2–3 sec each) of individual signs were selected for each type of stimulus where the sets were matched for age of acquisition (AoA), familiarity, iconicity, and complexity as explained below. BSL-only signs and cognates were initially drawn from Vinson, Cormier, Denmark, Schembri, and Vigliocco (2008), who provide a catalogue of BSL signs ranked by 30 deaf signers with respect to AoA, familiarity, and iconicity. A set of SSL signs was selected from the SSL Dictionary (Hedberg et al., 2005), where all phonologically contrasting handshapes were included in the sample. All of the SSL signs were possible signs in BSL, but none were existing BSL lexical signs. Nonsigns were created by deaf native signers using a range of handshapes, locations, and movement patterns. Most of these nonsigns had previously been used in behavioral studies (Orfanidou, Adam, Morgan, & McQueen, 2010; Orfanidou et al., 2009); an additional set was created specifically for the current study. All nonsigns violated phonotactic rules of BSL and SSL or were made of nonoccurring combinations of parameters, including (a) two active hands performing symmetrical movements but with different handshapes; (b) compound-type nonsigns having two locations on the body but with movement from the lower location to the higher location (instead of going from the higher to the lower location1); (c) nonoccurring or unusual points of contact on the signer's body (e.g., occluding the signer's eye or the inner side of the upper arm); (d) nonoccurring or unusual points of contact between the signer's hand and the location (e.g., handshape with the index and middle finger extended, but contact only between the middle finger and the body); nonoccurring handshapes. For BSL-only signs and cognates, AoA, familiarity, and iconicity ratings were obtained from Vinson et al. (2008). Complexity ratings were obtained from two deaf native BSL signers. For SSL stimuli, two deaf native signers of SSL ranked all items for AoA, familiarity, iconicity, and complexity according to the standards used for the BSL sign rankings. For nonsigns, complexity ratings were obtained from deaf native BSL signers and deaf native SSL signers. For each video clip showing a single sign, participants were instructed to “Concentrate on the hand movements of the person in the video. For each video clip you should rate the sign on a scale of 0–4 as being simple or complex, where 0 = simple and 4 = complex. Each video clip will appear twice. You are supposed to make an instant judgment on whether the sign you are viewing seems simple or complex to YOU. Reply with your first impression. Do not spend more time on any one sign. Rate your responses on the sheet provided. Circle the figure YOU think best describes the sign in the video.” There were no significant differences between any two sets with respect to any of these features based on the average of the obtained ratings (p > .05 in all cases) with a single exception: Iconicity and familiarity of cognates were higher than that of BSL-only and SSL signs. This, however, is expected, because the term “cognate” is used here to refer to signs that share a common visual motivation (i.e., iconicity) and not to those signs that are historically related through a common linguistic ancestor, with the exception of country names. This group consists of signs that are known to be borrowed from their country of origin (i.e., the signs JAPAN in BSL and SSL are borrowed from the Japanese Sign Language). Mean duration of videos for each category was as follows (mean ± SEM): cognates = 2723 ± 24.0 msec; BSL = 2662 ± 30.6 msec; SSL = 2683 ± 25.2 msec; nonsigns = 2700 ± 27.3 msec. There were no significant differences between any two sets with respect to duration (p > .05 in all cases).

Participants performed monitoring tasks in which certain handshapes and locations were cued (see below). There were six different handshape cues and six different location cues (see Figure 1, bottom). Some handshape cues were constituted by collapsing across phonetically different handshapes, which were allophones of a single handshape (i.e., without a change in meaning in either BSL or SSL). Location cues were selected to reflect the natural distribution of signs across signing space: Chin, cheek, and neck are small areas but are close to the focus of gaze during reception of signing and were thus used as separate target positions; waist and chest are larger areas and farther from the focus of gaze. All cue pictures were still images extracted from video recordings made with the same parameters as the stimulus videos. Each handshape and location cue was used once for each stimulus type. Signs were chosen ensuring that all targets were present the same number of times for each stimulus type.

Figure 1. 

Stimuli and experimental design. Top: Diagrammatic representation of the experiment. Bottom: Cues: handshape (left) and location (right).

Figure 1. 

Stimuli and experimental design. Top: Diagrammatic representation of the experiment. Bottom: Cues: handshape (left) and location (right).

One of our main aims during the design of the stimuli was to avoid possible effects of familiarity with unknown signs due to repeated presentation of the stimuli set, hence the large number (48) of video clips per stimulus type. To achieve enough experimental power, each video clip had to be repeated once (it was not possible to enlarge the stimulus set while still controlling for AoA, familiarity, iconicity, complexity, number and type of targets in each stimulus type). To prevent possible effects of familiarity with the stimuli on task performance, stimulus was ordered such that no repetitions occurred across the different task types. The association between stimulus and tasks was counterbalanced across participants.

All stimulus items were recorded in a studio environment against a plain blue background using a digital high-definition camera. To ensure that any differences in activation between stimulus types were not driven by differences in sign production of a native versus foreign sign language (e.g., “accent”), signs were performed by a native user of German Sign Language, unfamiliar with either BSL or SSL. All items were signed with comparable ease, speed, and fluency and executed from a rest position to a rest position; signs were produced without any accompanying mouthing. Videos were edited with iMovieHD 6.0.3 and converted with AnyVideoConverter 3.0.3 to meet the constraints posed by the stimulus presentation software Cogent (www.vislab.ucl.ac.uk/cogent.php).

Stimuli were presented using Matlab 7.10 (The MathWorks, Inc., Natick, MA) with Cogent. All videos and images were presented at 480 × 360 pixels against a blue background. All stimuli were projected onto a screen hung in front of the magnet's bore; participants watched it through a mirror mounted on the headcoil.

Tasks and Experimental Design

Throughout the experiment, participants were asked to perform either a handshape or a location monitoring task. They were instructed to press a button with their right index finger when a sign occurred in a cued location or when they spotted the cued handshape as a part of a stimulus. This is a phoneme monitoring task (cf. Grosvald et al., 2012) for signers but can be performed as a purely perceptual matching task by nonsigners. Performance in the task was evaluated by calculating an adapted d′. Participants only pressed a button to indicate a positive answer (i.e., the presence of a particular handshape or a sign produced in the cued location). Therefore, we calculated hits and false positives from the instances in which the button presses were correct and incorrect (respectively). We then equated instances in which participants did not press the button as “no” answers and calculated correct rejections and misses from the situations in which the lack of response was correct and incorrect (respectively).

Stimuli of each type (BSL, cognates, SSL, and nonsigns) were presented in blocks. Prior to each block, a cue picture showed which handshape or location to monitor (Figure 1, top). In total, there were 12 blocks per stimulus type presented in a randomized order. Each block contained eight videos of the same type of stimulus. Videos were separated by an intertrial interval where a blank screen was displayed for 2–6 sec (4.5 sec average). Prior to the onset of each video, a fixation cross in the same spatial location as the model's chin was displayed for 500 msec. Participants were asked to fixate on the signer's chin, given that the lower face area corresponds to the natural focus of gaze in sign language communication (Agrafiotis, Canagarajah, Bull, & Dye, 2003). Between blocks, participants were presented a 15-sec baseline video of the still model with a yellow fixation cross on the chin (Figure 1, top). They were instructed to press the button when the cross changed to red. This vigilance task has previously been used as a baseline condition in fMRI studies (e.g., Capek et al., 2008). In subsequent instances in the manuscript, the term “baseline” will refer to this 15-sec period while the model was in a static position. This baseline condition is different from blank periods of no visual stimulation, which were also present in between blocks and videos, as described.

Each participant performed four scanning runs, each consisting of 12 blocks. To make it easier for participants to focus on one of the two types of monitoring tasks, each participant performed either two runs consisting exclusively of location tasks followed by two runs consisting of handshape tasks or vice versa. The order of the tasks and stimulus types was counterbalanced across participants, with no participant in the same experimental group encountering the stimuli in the same order.

Testing Procedure

Before the experiment, the tasks were explained to the participants in their preferred language (BSL or English), and written instructions were also provided in English. A short practice session, using different video clips from those used in the main experiment, ensured that the participants were able to solve both tasks.

During scanning, participants were given a button-box and instructed to press a button with their right index finger whenever they recognized a target during the monitoring tasks or when the baseline fixation cross changed color. There were two video cameras in the magnet's bore. One was used to monitor the participant's face and ensure they were relaxed and awake throughout scanning; the other monitored the participant's left hand, which was used by deaf signers for manual communication with the researchers between scans. A third video camera in the control room was used to relay signed instructions to the participant via the screen. Researchers communicated with deaf nonsigner participants through written English displayed on the screen; deaf nonsigner participants responded using speech. An intercom was used for communication with hearing participants. All volunteers were given ear protection.

After scanning, a recognition test was performed where all signed stimuli used in the experiment were presented outside the scanner to the deaf signers, and they were asked to indicate for each stimulus whether it was a familiar sign and, if so, to state its meaning. This procedure was used to ensure that all items were correctly categorized by each individual. Items not matching their assigned stimulus type were excluded from subsequent analyses for that individual.

Image Acquisition and Data Analysis

Images were acquired at the Birkbeck-UCL Centre for Neuroimaging, London, with a 1.5-T Siemens Avanto scanner (Siemens, Erlangen, Germany) and a 32-channel head coil. Functional imaging data were acquired using a gradient-echo EPI sequence (repetition time = 2975 msec, echo time = 50 msec, field of view = 192 × 192 mm) giving a notional resolution of 3 × 3 × 3 mm. Thirty-five slices were acquired to obtain whole-brain coverage without the cerebellum. Each experimental run consisted of 348 volumes taking approximately 17 min to acquire. The first seven volumes of each run were discarded to allow for T1 equilibration effects. An automatic shimming algorithm was used to reduce magnetic field inhomogeneities. A high-resolution structural scan for anatomical localization purposes (magnetization-prepared rapid acquisition with gradient echo, repetition time = 2730 msec, echo time = 3.57 msec, 1 mm3 resolution, 176 slices) was taken either at the end or in the middle of the session.

Imaging data were analyzed using Matlab 7.10 and Statistical Parametric Mapping software (SPM8; Wellcome Trust Centre for Neuroimaging, London, UK). Images were realigned, coregistered, normalized, and smoothed (8 mm FWHM Gaussian kernel) following SPM8 standard preprocessing procedures. Analysis was conducted by fitting a general linear model with regressors representing each stimulus type, task, baseline, and cue periods. For every regressor, events were modeled as a boxcar of the adequate duration, convolved with SPM's canonical hemodynamic response function and entered into a multiple regression analysis to generate parameter estimates for each regressor at every voxel. Movement parameters were derived from the realignment of the images and included in the model as regressors of no interest.

Contrasts for each experimental stimulus type and task (e.g., [BSL location > Baseline]) were defined individually for each participant and taken to a second-level analysis. To test for main effects and interactions, a full-factorial second-level whole-brain analysis was performed. The factors entered into the analysis were group (deaf signers, deaf nonsigners, hearing nonsigners), task (handshape, location), and stimulus type (BSL, SSL, cognates, nonsigns). Age and gender were included as covariates. Main effects and interactions were tested using specified t contrasts. Voxels are reported as x, y, z coordinates in accordance with standard brains from the Montreal Neurological Institute (MNI). Activations are shown at p < .001 or p < .005 uncorrected thresholds for display purposes, but they are only discussed if they reached a significance threshold of p < .05 (corrected) at peak or cluster level. Small volume corrections were applied if activations were found in regions where, given our literature review, we expected to find differences. If this correction was applied, we have specifically indicated it in the text.

Cognates were included in the experiment for crosslinguistic comparisons between BSL and SSL signers in a different report, and their classification as such is not relevant here. The only difference between BSL-only and cognates is their degree of iconicity and familiarity. We found no differences in neural activation due to differences in iconicity between BSL-only and cognates. Therefore, given that both sets of signs are part of the BSL lexicon, these types of stimuli were combined into a single class in the analyses and are referred to as BSL signs in the Results section.

RESULTS

Our study aimed to determine if neurocognitive mechanisms involved in processing sensorimotor characteristics of the sign language signal are differentially recruited for phonological processing and how these are modulated by the semantic and phonological structure of the stimuli. For this purpose, we first report the behavioral performance in the handshape and location tasks, identifying differences between tasks and stimuli that could be reflected in the neuroimaging results. We then show a conjunction of the neuroimaging results across all the groups, stimulus types, and tasks to identify the brain regions that were recruited for solving the tasks independently of stimulus properties, sign language knowledge, and hearing status. Group effects are reported after this to dissociate these from the subsequently reported main effects of Task, Stimulus types, and Interactions that specifically test our hypotheses.

Behavioral Results

Behavioral performance was measured using d′ and RTs (Table 3). A repeated-measures ANOVA with adapted d′ as the dependent variable and the factors Group (deaf signers, deaf nonsigners, hearing nonsigners), Task (handshape, location), and Stimulus type (BSL, SSL and nonsigns) resulted in no significant main effects or interactions: Stimulus type (F(2, 80) = 1.98, p = .14), Task (F(1, 40) = 1.72, p = .20), Group (F < 1, p = .52), Stimulus type × Task (F(2, 80) < 1, p = .65), Stimulus type × Group (F(4, 80) = 1.18, p = .32), Task × Group (F(2, 40) = 2.03, p = .14), three-way interaction (F(6, 120) = 1.20, p = .31).

Table 3. 

Behavioral Performance for the Handshape and Location Tasks

Deaf SignersDeaf OralHearing Nonsigners
RTSDd′SDRTSDd′SDRTSDd′SD
Handshape 
BSL 1.43 0.23 2.70 0.92 1.48 0.19 2.61 0.51 1.59 0.28 2.59 0.45 
SSL 1.43 0.29 2.60 0.69 1.42 0.22 2.61 0.68 1.58 0.30 2.57 0.68 
Nonsigns 1.69 0.31 2.54 0.64 1.60 0.17 2.62 0.60 1.63 0.25 2.38 0.67 
 
Location 
BSL 1.17 0.26 2.83 0.75 1.19 0.16 2.38 0.28 1.29 0.26 2.87 0.54 
SSL 1.23 0.26 3.03 0.79 1.23 0.14 2.48 0.71 1.34 0.27 2.82 0.63 
Nonsigns 1.44 0.20 2.80 0.63 1.36 0.10 2.51 0.32 1.51 0.22 2.54 0.65 
Deaf SignersDeaf OralHearing Nonsigners
RTSDd′SDRTSDd′SDRTSDd′SD
Handshape 
BSL 1.43 0.23 2.70 0.92 1.48 0.19 2.61 0.51 1.59 0.28 2.59 0.45 
SSL 1.43 0.29 2.60 0.69 1.42 0.22 2.61 0.68 1.58 0.30 2.57 0.68 
Nonsigns 1.69 0.31 2.54 0.64 1.60 0.17 2.62 0.60 1.63 0.25 2.38 0.67 
 
Location 
BSL 1.17 0.26 2.83 0.75 1.19 0.16 2.38 0.28 1.29 0.26 2.87 0.54 
SSL 1.23 0.26 3.03 0.79 1.23 0.14 2.48 0.71 1.34 0.27 2.82 0.63 
Nonsigns 1.44 0.20 2.80 0.63 1.36 0.10 2.51 0.32 1.51 0.22 2.54 0.65 

The table lists mean RTs and d′ for the handshape and location tasks, and each stimulus type, separately for each group.

A similar repeated-measures ANOVA with RT as the dependent variable showed a significant main effect of Stimulus type (F(2, 80) = 52.66, p < .001), a significant main effect of Task (F(1, 40) = 64.44, p < .001), and a significant interaction of Stimulus type × Group (F(4, 80) = 3.06, p = .021). The interaction of Stimulus type × Task (F(2, 80) = 2.74, p = .071) approached significance. There was no significant main effect of Group (F(2, 40) = 1.27, p = .29), no significant interaction of Task × Group (F(2, 40) < 1, p = .96), and no three-way interaction (F(4, 80) = 1.55, p = .19). Pairwise comparisons between stimulus types revealed that participants were significantly slower judging nonsigns than BSL (t(42) = 7.67, p < .001) and SSL (t(42) = 9.44, p < .001), but no significant difference was found between BSL and SSL (t(42) = 0.82, p = .40). They also showed that participants are significantly faster in the location task compared with the handshape task (t(42) = 7.93, p < .001). Pairwise comparisons investigating the interaction between Stimulus type × Group are presented in Table 4. The deaf signers group was significantly faster (p < .05, Bonferroni-corrected) than the hearing nonsigners group for BSL and SSL, but not for nonsigns. It should be noticed that the deaf nonsigners group was faster than the hearing nonsigners group also for BSL and SSL, but these differences do not survive correction for multiple comparisons. There was no significant difference in RT between the deaf signers and the deaf nonsigners groups.

Table 4. 

Least Significant Difference Pairwise Comparisons for RT Results for the Interaction Stimulus Type × Group

BSLSSLNonsigns
t(42)pt(42)pt(42)p
Deaf signers–Deaf oral 0.61 .54 0.039 .97 1.58 .12 
Deaf signers–Hearing nonsigners 3.12 .003* 2.94 .005* 0.13 .90 
Deaf oral–Hearing nonsigners 2.13 .04 2.65 .01 1.86 .07 
BSLSSLNonsigns
t(42)pt(42)pt(42)p
Deaf signers–Deaf oral 0.61 .54 0.039 .97 1.58 .12 
Deaf signers–Hearing nonsigners 3.12 .003* 2.94 .005* 0.13 .90 
Deaf oral–Hearing nonsigners 2.13 .04 2.65 .01 1.86 .07 

Least significant difference pairwise comparisons for RT results. The table shows absolute t values.

*Values surviving significance at p < .0055 (uncorrected), which is equivalent to p = .05 corrected for multiple comparisons (Bonferroni).

fMRI Results

Conjunction

Figure 2 shows the areas that were recruited to perform both tasks in all groups, collapsing across stimulus type and task. Activations were observed bilaterally in middle occipital regions, extending anteriorly and ventrally to the inferior temporal cortex and the fusiform gyrus and dorsally toward superior occipital regions and the inferior parietal lobe. Activations were also observed in the middle and superior temporal cortex, the superior parietal lobe (dorsal to the postcentral gyrus), and the IFG (pars opercularis). See Table 5.

Figure 2. 

Conjunction of all tasks and all stimulus types in all of the experimental groups (deaf signers, deaf nonsigners, hearing nonsigners). The figure shows the significant activations (p < .001, uncorrected) for the conjunction of the contrasts of each stimulus type and task against the baseline condition.

Figure 2. 

Conjunction of all tasks and all stimulus types in all of the experimental groups (deaf signers, deaf nonsigners, hearing nonsigners). The figure shows the significant activations (p < .001, uncorrected) for the conjunction of the contrasts of each stimulus type and task against the baseline condition.

Table 5. 

Peak Coordinates for Conjunction Analysis

NamePeak Voxel
p (Corr)Z Scorexyz
Middle occipital cortex <.0001 >8.00 −27 −91 
<.0001 >8.00 27 −91 10 
Calcarine sulcus .0005 5.51 −15 −73 
.0010 5.38 12 −70 10 
Middle temporal gyrus <.0001 >8.00 −45 −73 
<.0001 >8.00 51 −64 
Superior parietal lobule .0039 5.10 21 −67 52 
Inferior parietal lobule <.0001 6.55 −30 −43 43 
.0001 5.75 39 −40 55 
IFG (pars opercularis) <.0001 6.48 −51 40 
.0009 5.39 48 11 22 
Insula .0461 4.53 33 29 
NamePeak Voxel
p (Corr)Z Scorexyz
Middle occipital cortex <.0001 >8.00 −27 −91 
<.0001 >8.00 27 −91 10 
Calcarine sulcus .0005 5.51 −15 −73 
.0010 5.38 12 −70 10 
Middle temporal gyrus <.0001 >8.00 −45 −73 
<.0001 >8.00 51 −64 
Superior parietal lobule .0039 5.10 21 −67 52 
Inferior parietal lobule <.0001 6.55 −30 −43 43 
.0001 5.75 39 −40 55 
IFG (pars opercularis) <.0001 6.48 −51 40 
.0009 5.39 48 11 22 
Insula .0461 4.53 33 29 

The table shows the peak of activations for a conjunction analysis between groups, collapsing across tasks and stimulus type. L = left; R = right. Corr: p < .05, FWE.

Effect of Group

To evaluate the effects driven by sign language experience and hearing status, which were independent of task and stimulus type, we collapsed results across all tasks and stimulus types and then compared the activations between groups. Figure 3A shows stronger bilateral activations in superior temporal cortex (STC) in the group of deaf signers, compared to the groups of deaf nonsigners and hearing nonsigners (Table 6; this result was previously published in Cardin et al., 2013). Figure 4 shows that all the stimulus types and tasks activated the STC bilaterally over the baseline. To determine if the two groups of nonsigners (hearing and deaf) were using different strategies or relying differentially on perceptual processing, we conducted a series of comparisons to identify activations that were present exclusively in deaf nonsigners and hearing nonsigners (Table 6). Figure 3B shows that hearing nonsigners recruited occipital and superior parietal regions across tasks and stimulus types. This result is observed when hearing nonsigners are compared to both deaf signers and deaf nonsigners (using a conjunction analysis), demonstrating that this effect is driven by the difference in hearing status between the groups and not by a lack of sign language knowledge. Figure 3C shows a stronger focus of activity in the posterior middle temporal gyrus in the deaf nonsigners group. This effect was present bilaterally, but only the left hemisphere cluster was statistically significant (p < .05 corrected at peak level).

Figure 3. 

Effect of group. (A) Positive effect of deaf signers. The figure shows the conjunction of the contrasts [deaf signers > hearing nonsigners] and [deaf signers > hearing nonsigners]. This effect has been reported in Cardin et al. (2013). (B) Positive effect of hearing nonsigners. The figure shows the conjunction of the contrasts [hearing nonsigners > deaf signers] and [hearing nonsigners > deaf nonsigners]. (C) Positive effect of deaf nonsigners. The figure shows the conjunction of the contrasts [deaf nonsigners > deaf signers] and [deaf nonsigners > hearing nonsigners]. Activations are shown at p < .005 (uncorrected). DS = deaf signers group; HN = hearing nonsigners group; DN = deaf nonsigners group.

Figure 3. 

Effect of group. (A) Positive effect of deaf signers. The figure shows the conjunction of the contrasts [deaf signers > hearing nonsigners] and [deaf signers > hearing nonsigners]. This effect has been reported in Cardin et al. (2013). (B) Positive effect of hearing nonsigners. The figure shows the conjunction of the contrasts [hearing nonsigners > deaf signers] and [hearing nonsigners > deaf nonsigners]. (C) Positive effect of deaf nonsigners. The figure shows the conjunction of the contrasts [deaf nonsigners > deaf signers] and [deaf nonsigners > hearing nonsigners]. Activations are shown at p < .005 (uncorrected). DS = deaf signers group; HN = hearing nonsigners group; DN = deaf nonsigners group.

Table 6. 

Group Effects

Group EffectNamePeak Voxel
p (Corr)Z Scorexyz
Deaf signers Superior temporal cortex <.001 6.19 51 −25 
<.001 5.49 −60 −13 −2 
Hearing nonsigners Middle temporal gyrus .001 5.37 −45 −67 16 
.038 4.58 48 −58 13 
Middle occipital cortex .004 5.11 −45 −79 19 
Deaf oral Middle temporal gyrus .003 5.17 −57 −55 −2 
Group EffectNamePeak Voxel
p (Corr)Z Scorexyz
Deaf signers Superior temporal cortex <.001 6.19 51 −25 
<.001 5.49 −60 −13 −2 
Hearing nonsigners Middle temporal gyrus .001 5.37 −45 −67 16 
.038 4.58 48 −58 13 
Middle occipital cortex .004 5.11 −45 −79 19 
Deaf oral Middle temporal gyrus .003 5.17 −57 −55 −2 

The table shows the peak of activations for the main effect of each group, collapsing across tasks and stimulus type. L = left; R = right. Corr: p < .05, FWE.

Figure 4. 

The superior temporal cortex in deaf signers is activated by potentially communicative manual actions, independently of meaning, phonological structure, or task. The bar plot shows the effect sizes, relative to baseline, for the peak voxels in the superior temporal cortex for the conjunction of the contrasts [deaf signers > hearing nonsigners] and [deaf signers > deaf nonsigners] across all stimulus types and tasks. Bar represents means ± SEM.

Figure 4. 

The superior temporal cortex in deaf signers is activated by potentially communicative manual actions, independently of meaning, phonological structure, or task. The bar plot shows the effect sizes, relative to baseline, for the peak voxels in the superior temporal cortex for the conjunction of the contrasts [deaf signers > hearing nonsigners] and [deaf signers > deaf nonsigners] across all stimulus types and tasks. Bar represents means ± SEM.

Effect of Task

We hypothesized that different perceptual and motor brain regions would be recruited for the processing of handshape and location independently of participants' hearing status and sign language knowledge. Specifically, we expected dorsal visual areas, medial pFC, ACC, and the precuneus to be more active during the monitoring of location, and ventral visual areas, superior parietal lobule, the intraparietal sulcus, and motor and premotor regions to be more active while monitoring handshape. To test this, we compared the handshape task to the location task, collapsing across materials and groups. As can be seen in Figure 5 and Table 7, when evaluating the contrast [handshape > location], the handshape task activated more strongly prestriate regions and visual ventral areas in the fusiform gyrus and the inferior temporal gyrus, but also parietal regions along the intraparietal sulcus, the IFG (anteriorly and dorsal to area 45), and the dorsal portion of area 44. In contrast, the comparison [location > handshape] shows that the location task recruited more strongly dorsal areas such as the angular gyrus and the precuneus, in addition to the medial pFC, frontal pole, and middle frontal gyrus.

Figure 5. 

Monitoring of phonological parameters in sign language recruits different perceptual networks, but the same linguistic network. Top: The figure shows the results for the contrast [handshape > location] (top left) and [location > handshape] (top right) across all groups of participants. Bottom: The same contrasts are shown overlapped onto brain slices of SPM8's MNI standard brain (bottom). All results at p < .005 (uncorrected).

Figure 5. 

Monitoring of phonological parameters in sign language recruits different perceptual networks, but the same linguistic network. Top: The figure shows the results for the contrast [handshape > location] (top left) and [location > handshape] (top right) across all groups of participants. Bottom: The same contrasts are shown overlapped onto brain slices of SPM8's MNI standard brain (bottom). All results at p < .005 (uncorrected).

Table 7. 

Task Effects

NamePeak Voxel
p (Corr)Z Scorexyz
[Handshape > Location] 
Ventral occipito-temporal cortex <.0001 >8.00 −18 −85 −8 
Inferior occipital cortex <.0001 >8.00 −15 −91 
<.0001 7.76 −75 
Inferior parietal lobule <.0001 7.24 −48 −34 43 
Postcentral gyrus <.0001 7.78 48 −28 49 
Precentral gyrus <.0001 >8.00 −45 31 
<.0001 7.68 48 31 
Anterior IFG <.0001 5.94 −39 35 16 
.0014 5.31 45 35 16 
Cerebellum .0161 4.78 −70 −20 
 
[Location > Handshape] 
Angular gyrus <.0001 >8.00 −42 −76 31 
<.0001 >8.00 48 −70 31 
Precuneus .0001 5.80 −12 −58 19 
<.0001 7.68 −61 58 
<.0001 6.55 15 −58 22 
pFC .0153 4.79 18 62 
Frontal pole .0227 4.70 59 
Middle frontal gyrus .0193 4.74 30 32 46 
NamePeak Voxel
p (Corr)Z Scorexyz
[Handshape > Location] 
Ventral occipito-temporal cortex <.0001 >8.00 −18 −85 −8 
Inferior occipital cortex <.0001 >8.00 −15 −91 
<.0001 7.76 −75 
Inferior parietal lobule <.0001 7.24 −48 −34 43 
Postcentral gyrus <.0001 7.78 48 −28 49 
Precentral gyrus <.0001 >8.00 −45 31 
<.0001 7.68 48 31 
Anterior IFG <.0001 5.94 −39 35 16 
.0014 5.31 45 35 16 
Cerebellum .0161 4.78 −70 −20 
 
[Location > Handshape] 
Angular gyrus <.0001 >8.00 −42 −76 31 
<.0001 >8.00 48 −70 31 
Precuneus .0001 5.80 −12 −58 19 
<.0001 7.68 −61 58 
<.0001 6.55 15 −58 22 
pFC .0153 4.79 18 62 
Frontal pole .0227 4.70 59 
Middle frontal gyrus .0193 4.74 30 32 46 

The table shows the peak of activations for the main effect of each task, collapsing across groups and stimulus type. L = left; R = right. Corr: p < .05, FWE.

To determine if phonological processing in sign language is specifically related to the sensorimotor characteristics of the language signal, we evaluated differential processing of these parameters in each of our groups using a Group × Task interaction. For example, if visual ventral areas are recruited differentially for the linguistic processing of handshape, we would expect to find differences in the activations between the handshape and location tasks in the deaf signers group that were not present in the other two groups. However, if phonological processing of handshape and location was independent of the sensorimotor characteristics of the input signal, we would expect each of them recruiting language processing areas (such as the STC) in the group of deaf signers, but not differentially. As shown in Figures 3A and 4, both handshape and location tasks activated more strongly bilateral STC regions in the deaf signers group than in the other two groups. However, a Group × Task interaction analysis ([deaf signers (handshape > location) ≠ deaf nonsigners (handshape > location)] & [deaf signers (handshape > location) ≠ hearing nonsigners (handshape > location)]) that specifically tested for differential handshape- or location-related activity in deaf signers resulted in no significantly active voxel at p < .05 corrected at peak or cluster level.

Effect of Stimulus Type

Semantics

To determine if the neural mechanisms underpinning phoneme monitoring are influenced by the participant's ability to access the meaning of the monitored stimulus, we evaluated the differential effect of stimuli with similar phonology, but from a known (BSL) or unknown (SSL) language. We first evaluated the contrasts [BSL > SSL] and [SSL > BSL] in the groups of nonsigners to exclude any differences due to visuospatial characteristics of the stimuli, rather than linguistic ones. There was no significant effect of these two contrasts in either of the groups of nonsigners. The contrasts [BSL > SSL] and [SSL > BSL] also resulted in no significant (p < .05 corrected at peak or cluster level) effects in deaf signers.

Phonological structure

To evaluate if the neural mechanisms underpinning phoneme monitoring are influenced by the phonological structure of natural language even when that structure has no linguistic significance, nonsigns were compared to all the other sign stimuli (BSL and SSL, which have phonologically acceptable structure). Given the lack of an effect of semantics, differences across all sign stimuli will be driven by differences in phonological structure and not semantics. We favored a comparison of nonsigns to all the other stimulus types because an effect due to differences in phonological structure in the stimuli should distinguish the nonsigns also from BSL and not only from SSL. No significant (p < .05 corrected at peak or cluster level) activations were found for the contrast [signs > nonsigns]. However, there was a main effect of [nonsigns > signs] across groups and tasks (Figure 6A), indicating that this was a general effect in response to this type of stimuli and not a specific one related to linguistic processing (Table 8). Significant activations (p < .05 corrected at peak or cluster level) were observed in an action observation network including lateral occipital regions, intraparietal sulcus, superior parietal lobe, SMG, IFG (pars opercularis), and thalamus.

Figure 6. 

Nonsigns differentially activate action observation and phonological processing areas. Top: The figure shows the results of the contrast [nonsigns > (BSL + SSL)] in all groups of participants (p < .005, uncorrected). The bar plot shows the effect sizes relative to baseline for the most significant clusters (inferior parietal sulcus, IPS). Bars represent means ± SEM. Bottom: Interaction effect. The figure shows the results of the Group × Stimulus type interaction, where the results of the [nonsigns > (BSL + SSL)] contrast in deaf signers are compared to those in the deaf nonsigners and hearing nonsigners (p < .005, uncorrected). The contrast description is: [deaf signers (nonsigns > (BSL + SSL)) > (deaf nonsigners & hearing nonsigners) (nonsigns > (BSL + SSL))]. Bar plot showing effect sizes from the SMG (details as described above). The brain slices show the results for the contrast [nonsigns > (BSL + SSL)] in each of the experimental groups and the result of the Group × Stimulus type interaction. DS = deaf signers group; HN = hearing nonsigners group; DN = deaf nonsigners group.

Figure 6. 

Nonsigns differentially activate action observation and phonological processing areas. Top: The figure shows the results of the contrast [nonsigns > (BSL + SSL)] in all groups of participants (p < .005, uncorrected). The bar plot shows the effect sizes relative to baseline for the most significant clusters (inferior parietal sulcus, IPS). Bars represent means ± SEM. Bottom: Interaction effect. The figure shows the results of the Group × Stimulus type interaction, where the results of the [nonsigns > (BSL + SSL)] contrast in deaf signers are compared to those in the deaf nonsigners and hearing nonsigners (p < .005, uncorrected). The contrast description is: [deaf signers (nonsigns > (BSL + SSL)) > (deaf nonsigners & hearing nonsigners) (nonsigns > (BSL + SSL))]. Bar plot showing effect sizes from the SMG (details as described above). The brain slices show the results for the contrast [nonsigns > (BSL + SSL)] in each of the experimental groups and the result of the Group × Stimulus type interaction. DS = deaf signers group; HN = hearing nonsigners group; DN = deaf nonsigners group.

Table 8. 

Peak Activations for the Contrast [Nonsigns > Signs]

NamePeak Voxel
p (Corr)Z Scoresxyx
Intraparietal sulcus <.001 6.01 −36 −43 46 
.003 5.12 36 −46 49 
SMG .001 5.49 −51 −31 40 
.007 4.96 42 −37 49 
Superior parietal lobule .031 4.63 −18 −67 52 
.002 5.21 21 −61 52 
Thalamus .029 4.65 18 −28 
Middle occipital cortex .002 5.19 −30 −82 22 
.044 4.60 39 −79 16 
IFG (pars opercularis) .031 4.62 51 31 
NamePeak Voxel
p (Corr)Z Scoresxyx
Intraparietal sulcus <.001 6.01 −36 −43 46 
.003 5.12 36 −46 49 
SMG .001 5.49 −51 −31 40 
.007 4.96 42 −37 49 
Superior parietal lobule .031 4.63 −18 −67 52 
.002 5.21 21 −61 52 
Thalamus .029 4.65 18 −28 
Middle occipital cortex .002 5.19 −30 −82 22 
.044 4.60 39 −79 16 
IFG (pars opercularis) .031 4.62 51 31 

The table shows the peak of activations for the contrast [Nonsigns > Signs], collapsing across groups and tasks. L = left; R = right. Corr: p < .05, FWE.

To determine if there was any region that was recruited differentially in deaf signers, which would indicate modulation of the phoneme monitoring task by phonological structure, we evaluated the interaction between groups and stimulus types [deaf signers (nonsigns > signs)] > [deaf nonsigners + hearing nonsigners (nonsigns > signs)]. Results from this interaction show significant activations (p < .005, uncorrected) in bilateral SMG, anterior to parieto-temporal junction (Figure 6, bottom; Table 9). Because the SMG was one of the regions in which we predicted an effect in phonological processing, we applied a small volume (10 mm) correction to this activation, which resulted in significance at p < .05. Brain slices in Figure 6B show that uncorrected (p < .005) activations in this region of the SMG are present only in the deaf signers group and not in either deaf nonsigners or hearing nonsigners groups.

Table 9. 

Peak Voxels for the Group × Stimulus Type Interaction

NamePeak Voxel
p (Unc)Z Scorexyz
SMG .0002 3.47 −51 −34 25 
.0012 3.03 54 −28 22 
NamePeak Voxel
p (Unc)Z Scorexyz
SMG .0002 3.47 −51 −34 25 
.0012 3.03 54 −28 22 

This table shows results from the contrast [deaf signers (nonsigns > signs)] > [deaf nonsigners + hearing nonsigners (nonsigns > signs)]. L = left; R = right; unc = uncorrected.

Interaction between Task and Stimulus Type

It is possible that phonological processing in sign language is specifically related to the sensorimotor characteristics of the language signal only when participants can access meaning in the stimuli. To evaluate if handshape and location were processed differently for stimuli with different semantic and phonological structure, we assessed the interactions between Task and Stimulus type in the deaf signers group. No significant interactions were found (p < .05 corrected at peak or cluster level).

DISCUSSION

Our study characterized the neural processing of phonological parameters in visual language stimuli with different levels of linguistic structure. Our aim was to determine if the neural processing of phonologically relevant parameters is modulated by the sensorimotor characteristics of the language signal. Here we show that handshape and location are processed by different sensorimotor areas; however, when linguistic information is extracted, both these phonologically relevant parameters of sign language are processed in the same language regions. Semantic content does not seem to have an influence on phoneme monitoring in sign language, but phonological structure does. This was reflected by nonsigns causing a stronger activation of the SMG, an area involved in phonological function, only in deaf signers; this suggests that neural demands for linguistic processing are higher when stimuli are less coherent or have a less familiar structure. Our results also show that the identity of the brain regions recruited for the processing of signed stimuli depends on participants' hearing status and their sign language knowledge: Differential activations were observed in the superior temporal cortex for deaf signers, in posterior middle temporal gyrus for deaf nonsigners, and in occipital and parietal regions for hearing nonsigners. Furthermore, nonsigns also activated more strongly an action observation network in all participants, independently of their knowledge of sign language, probably reflecting a general increase in processing demands on the system.

The Superior Temporal Cortex Is Activated in Deaf Signers for the Monitoring of Handshape and Location, Independently of the Linguistic Content of the Stimuli

Monitoring handshape and location recruited bilateral STC in deaf signers, but not in either the hearing or deaf nonsigners. In a previous report (Cardin et al., 2013), we showed that activations elicited by sign language stimuli in the left STC of congenitally deaf individuals have a linguistic origin and are shaped by sign language experience, whereas, in contrast, the right STC shows activations assigned to both linguistic and general visuospatial processing, the latter being an effect of life-long plastic reorganization due to sensory deprivation. Here we extend these findings by showing that deaf native signers, but not the other groups, recruit the right and left STC for the processing of manual actions with potential communicative content, independently of the lack of meaning or the violation of phonological rules. This is in agreement with previous literature showing that the left IFG and middle and superior temporal regions are activated during observation of meaningless gestural strings (MacSweeney et al., 2004) or American Sign Language (ASL) pseudosigns (Emmorey, Xu, & Braun, 2011; Buchsbaum et al., 2005). The direct comparison of groups demonstrates that the effect in regions differentially recruited in deaf signers is due to sign language knowledge and not due to differences in hearing status. These results may seem at odds with MacSweeney et al. (2004), where similar neural responses were found for nonsigning groups in temporal cortices. However, given that signing and nonsigning groups were not directly contrasted in that study, it was not clear whether signers may have recruited perisylvian language regions to a greater extent.

Handshape and Location Are Processed by Different Perceptual Networks, but the Same Linguistic Network

SL phonology relates to patterning of handshape and hand location in relation to the body and hand movement with regard to the actively signing hand (Emmorey, 2002). However, although the semantic level of language processing can be understood in similar ways for sign and speech, the phonological level of language processing may be specifically related to the sensorimotor characteristics of the language signal. Although it has been shown that the neural network supporting phonological processing is to some extent supramodal (MacSweeney, Waters, et al., 2008), the processing of different phonological components, such as handshape and location, could recruit distinct networks, at least partially. Here we show that different phonological components of sign languages are indeed processed by separate sensorimotor networks, but that both components recruit the same language-processing regions when linguistic information is extracted. In deaf signers, the extraction of handshape and hand location in sign-based material did evoke implicit linguistic processing mechanisms, shown by the specific recruitment of STC for each of these tasks only in this group. However, this neural effect was not reflected on performance. Furthermore, the interaction between group and task did not result in any significantly activated voxel, suggesting that phonological processing in SL is not related to specific sensorimotor characteristics of the signal. Differences between the handshape and the location tasks were observed in all the experimental groups, independently of their SL knowledge or hearing status, suggesting that the differences are related to basic perceptual processing of the stimuli or task-specific demands. Specifically, extracting handshape recruits ventral visual regions involved in object recognition, such as the fusiform gyrus and the inferior temporal gyrus, and dorsal parietal regions involved in mental rotation of objects (Bracci et al., 2010; Op de Beeck et al., 2010; Wilson & Farah, 2006; Koshino, Carpenter, Keller, & Just, 2005). The location task resulted in the activation of dorsal areas such as the angular gyrus and the precuneus, as well as prefrontal areas, involved in the perception of space, localization of body parts, self-monitoring, and reorientation of spatial attention (Chen, Weidner, Vossel, Weiss, & Fink, 2012; Felician et al., 2009; Kelley et al., 2002).

The significant difference in RTs between tasks across groups suggests that distinct neural activations may be due, at least partly, to differences in task difficulty or cognitive demands. The cognitive demands of the handshape task are greater than those of the location task. Although the handshape task involves determining which hand to track and resolving handshape, even when partially occluded, the location task could be solved simply by allocating attention to the cued region of the field of view. As a reflection of these differences, participants in all groups were significantly faster at detecting location targets compared to handshape targets. In agreement with the observed behavioral effect, stronger activations were found for the handshape task in the inferior parietal lobule and the IFG, which are regions that are involved in cognitive control and where activation correlates with task difficulty (Cole & Schneider, 2007). Furthermore, activity in the precuneus, which was more active in the location task, has been shown to correlate negatively with task difficulty (Gilbert, Bird, Frith, & Burgess, 2012).

The fact that handshape and location did not elicit different activations in language-processing areas in deaf signers does not exclude the possibility that these two features contribute differently to lexical access. In a previous ERP study, Gutiérrez, Müller, et al. (2012) found differences in the neural signature relating to handshape and location priming. An interesting possibility is that the processing of handshape and location do indeed have a different role in lexical access, as postulated by Gutiérrez et al., but are processed within the same linguistic network, with differences in timing (and role in lexical access) between handshape and location arising as a reflection of different delays in internetwork connectivity between the perceptual processing of these phonological parameters and its linguistic one.

Phoneme Monitoring Is Independent of Meaning

Our results show no difference in the pattern of brain activity of deaf signers for signs that belonged to their own sign language (BSL) and were thus meaningful and those that belonged to a different sign language (SSL) and were thus not meaningful. This result is in agreement with Petitto et al. (2000), who found no differences in the pattern of activations observed while signing participants were passively viewing ASL signs or “meaningless sign-phonetic units that were syllabically organized into possible but nonexisting, short syllable strings” (equivalent to our SSL stimuli). Our results are also at least partially in agreement with those of Emmorey et al. (2011), who did not observe regions recruited more strongly for meaningful signs compared to pseudosigns (equivalent to our SSL stimuli), and Husain, Patkin, Kim, Braun, and Horwitz (2012), who only found a stronger activation for ASL compared to pseudo-ASL in the cuneus (26, −74, 20). The cuneus is the region mostly devoted to visual processing, and Husain et al.'s (2012) result could be due to basic visual feature differences between the stimuli, given that this contrast was not evaluated in an interaction with a control group. However, the lack of differential activations between BSL and SSL stimuli is at odds with other signed language literature (Emmorey et al., 2011; MacSweeney et al., 2004; Neville et al., 1998). In the study of MacSweeney et al. (2004), the differences between stimuli were not purely semantic, and the effects of other factors, such as phonology, cannot be ruled out.

Another source of discrepancy could be the nature of the tasks. Because the main goal of this study was to dissociate perceptual and linguistic processing of handshape and location, our tasks were chosen so that both signers and nonsigners could perform at comparable levels, not demanding explicit semantic judgements of the stimuli. In Emmorey et al. (2011), participants had to view stimuli passively, but knew they were going to be asked questions about stimulus identity after scanning. In Neville et al. (1998), participants performed recognition tests at the end of each run, and in MacSweeney et al. (2004), participants had to indicate or “guess” which sentences made sense. Thus, the tasks used in all three of these studies required the participants to engage in semantic processing. The contrast between the results of this study and previous ones may be understood in terms of levels of processing whereby deeper memory encoding is engendered by a semantic task, compared to the shallow memory encoding engendered by a phonological task (Craik & Lockhart, 1972), resulting also in stronger activations in the former. Recent work has identified such an effect for sign language (Rudner et al., 2013). It has also been suggested that semantic and lexical processing are ongoing, automatic processes in the human brain and that differences in semantic processing are only observed when task demands and reallocation of attention from internal to external processes are engaged (see Binder, 2012, for a review). If semantic processing is a default state, it would be expected that, when the task does not require explicit semantic retrieval and can be solved by perceptual and phonological mechanisms, as in our study, the processing of single signs of a known and unknown language would not result in any difference in overall semantic processing.

The lack of differences when comparing meaningful and meaningless signs could also be due to the strong relationship between semantics and phonology in sign languages. Although the SSL signs and the nonsigns do not have explicit meaning for BSL users, phonological parameters such as location, handshape, and movement are linked to specific types of meaning. For example, signs in BSL produced around the head usually relate to mental or cognitive processes; those with a handshape in which only the little finger is extended usually have a negative connotation (Sutton-Spence & Woll, 1999). This, added to the fact that deaf people often must communicate with hearing peers who do not know sign language and that communicative gestures can be identified as such (Willems & Hagoort, 2007), could explain why there is no difference between stimuli with and without semantic content—meaning will be extracted (whether correct or not), at least to a certain extent, from any type of sign.

Nonsigns Differentially Activate Action Observation and Phonological Processing Areas

Monitoring nonsigns resulted in higher activations in regions that are part of an action–observation network in the human brain (see Corina & Knapp, 2006, for a review), including middle occipital regions, intraparietal sulcus, SMG, IFG (pars opercularis), and thalamus. This effect was observed in all groups, independently of sign language knowledge and hearing status, suggesting that it is due to inherent properties of the stimuli, such as the articulations of the hand and arm and the visual image they produce, and not due simply to being unusual or to violations of linguistic structure. These higher activations in response to nonsigns could be due to more complex movements and visuospatial integration for such stimuli. This will in turn make these signs more difficult to decode, increasing the processing demands in the system, and potentially recruiting additional frontal and parietal areas to aid in the disambiguation of the stimuli. In support of our results, a previous study (Costantini et al., 2005) showed stronger activations in posterior parietal cortex for the observations of impossible manual actions compared to possible ones. The authors suggested that this was due to higher demands on the sensorimotor transformations between sensory and motor representations that occur in this area. Behaviorally, performance in the tasks was slower for all groups with nonsigns compared to BSL and SSL, supporting the idea that overall higher demands were imposed to the system.

We also observed that nonsigns caused a stronger activation, only in deaf signers, in the SMG. This effect suggests a modulation of phoneme monitoring by phonological structure of the signal and corroborates the role of this area in phonological processing of signed (MacSweeney, Waters, et al., 2008; Emmorey et al., 2002, 2007; Emmorey, Grabowski, et al., 2003; MacSweeney, Woll, Campbell, Calvert, et al., 2002; Corina et al., 1999) and spoken language (Sliwinska, Khadilkar, Campbell-Ratcliffe, Quevenco, & Devlin, 2012; Hartwigsen et al., 2010). It also demonstrates that an increase in processing demands when stimuli are less coherent is seen not only at a perceptual level but also at a linguistic one. In short, the interaction effect observed in bilateral SMG suggests that stimuli contravening the phonotactics of sign languages exert greater pressure on phonological mechanisms. This is in agreement with previous studies of speech showing that the repetition of nonwords composed of unfamiliar syllables results in higher activations predominantly in the left frontal and parietal regions when compared to nonwords composed of familiar syllables (Moser et al., 2009). The specific factor causing an increase in linguistic processing demands in SMG is not known. Possibilities include more complex movements, increased visuospatial integration demands, less common motor plans, or transitions between articulators. All these may also be responsible for the increase in activity in the action observation network, impacting as well phonological processing in the SMG.

Overall, the fact that violations of phonological rules result in higher demands on the system, independently of previous knowledge of the language, suggests that the phonological characteristics of a language may arise partly as a consequence of more efficient neural processing for the perception and production of the language components.

Posterior Middle Temporal Gyrus Is Recruited More Strongly in Deaf Nonsigners while Processing Dynamic Visuospatial Stimuli

One of the novelties of our study is the introduction of a group of deaf nonsigners individuals as a control group, which allows us to make a comparison between knowing and not knowing a sign language, within the context of auditory deprivation. Our results show that deaf nonsigners recruited more strongly a bilateral region in posterior middle temporal gyrus, when compared to both deaf signers and hearing nonsigners. Given that the stimuli had no explicit linguistic content for the deaf nonsigners who had no knowledge of sign language, this result suggests that life-long exclusive use of the visual component of the speech signal in combination with auditory deprivation results in a larger involvement of this region in the processing of dynamic visuospatial stimuli. This region is known to be involved in the processing of biological motion, including that of hands, mouth, and eyes (Pelphrey, Morris, Michelich, Allison, & McCarthy, 2005; Puce, Allison, Bentin, Gore, & McCarthy, 1998). This includes instances of biological motion as part of a language or a potential communicative display, as it is recruited for the processing of speechreading and sign stimuli in both signers and nonsigners (Capek et al., 2008; MacSweeney, Woll, Campbell, McGuire, et al., 2002). It is likely that deaf nonsigners extract meaningful information from biological motion more often in their everyday life than hearing nonsigners, hence the significant difference between these groups. In particular, this is more likely to happen when they know that manual actions may contain meaning or have a communicative purpose, as is the case with signs. This is also consistent with the role of this region in semantic processing via visual and auditory stimulation (Visser, Jefferies, Embleton, & Lambon Ralph, 2012; De Zubicaray, Rose, & McMahon, 2011). Deaf nonsigners are likely to use visuospatial rather than linguistic processing to extract meaning, given their lack of knowledge of the language, and this may be the reason a greater activation of the posterior middle temporal gyrus bilaterally is found for this group. In support of this, MacSweeney et al. (2004) showed that, compared with Tic-Tac (a nonlinguistic manual code used by racecourse bookmakers to communicate odds), sign language stimuli resulted in stronger activations in areas involved in visual movement processing, including the posterior middle temporal gyrus, particularly in participants who do not have sign language representations, suggesting that they analyze these sequences as complex dynamic visuospatial displays.

Parieto-occipital Regions Are Recruited More Strongly in Hearing than in Deaf Individuals during Visuospatial Processing

Stronger activations in middle occipital and superior parietal regions were observed in the group of hearing nonsigners, when compared to both groups of deaf individuals. In a previous study, a similar effect was observed when comparing group effects in a study of the processing of emblems (meaningful hand gestures; Husain et al., 2012), in which hearing nonsigners recruited more strongly than deaf signers bilateral occipital regions and the left parietal cortex. However, it was not clear if this was due to differences in sign language knowledge or differences in auditory deprivation. Here we show that this effect is driven by auditory deprivation, given that it is observed when the group of hearing nonsigners is compared to both groups of deaf participants. In our previous study (Cardin et al., 2013), we showed that both groups of deaf participants recruit posterior and lateral regions of the right STC to process sign language stimuli, suggesting that the right STC has a visuospatial function in deaf individuals (see also Fine, Finney, Boynton, & Dobkins, 2005). In short, to solve the perceptual demands of the task and in comparison to the hearing nonsigners group, both groups of deaf individuals recruit the right STC more strongly and parieto-occipital regions to a lesser extent. Behaviorally, there was no significant difference between the groups of deaf individuals, but there was evidence that both performed faster than the group of hearing nonsigners for BSL and SSL. Thus, it is possible to hypothesize that, due to crossmodal plasticity mechanisms, the right STC in deaf individuals takes over some of the visuospatial functions that in hearing individuals are performed by parieto-occipital regions and aids the resolution of visuospatial tasks. In support of this, studies in congenitally deaf cats have shown that the auditory cortex reorganizes selectively to support specific visuospatial functions, resulting in enhanced performance in corresponding behavioral tasks (Lomber, Meredith, & Kral, 2010).

Summary

To conclude, we show that the linguistic processing of different phonological parameters of sign language is independent from the sensorimotor characteristics of the language signal. Handshape and location are processed by separate networks, but this is exclusively at a perceptual or task-related level, with both components recruiting the same areas at a linguistic level. The neural processing of handshape and location was not influenced by the semantic content of the stimuli. Phonological structure did have an effect in the behavioral and neuroimaging results, with RTs for nonsigns being slower and stronger activations found in an action observation network in all participants and in the SMG exclusively in deaf signers. These results suggest an increase in processing demands when stimuli are less coherent both at a perceptual and at a linguistic level. Given that unusual combinations of phonological parameters or violations of phonological rules result in higher demands on the system, independently of previous knowledge of the language, we suggest that the phonological characteristics of a language may arise as a consequence of more efficient neural processing for the perception and production of the language components.

Acknowledgments

This study was funded by the Riksbankens Jubileumsfond (P2008-0481:1-E), the Swedish Council for Working Life and Social Research (2008-0846), and the Swedish Research Council (Linnaeus Centre HEAD), and by grants from the Economic and Social Research Council of Great Britain (RES-620–28–6001; RES-620-28-6002) to the Deafness Cognition and Language Research Centre. We would like to thank Mischa Cooke, Lena Davidsson, Anders Hermansson, Ramas Rentelis, Lilli Risner, and Guiping Xu for their help with the recruitment of participants and the acquisition of MRI data. We specially thank all the deaf and hearing participants who took part in the study.

Reprint requests should be sent to Velia Cardin, Deafness, Cognition and Language Research Centre, Department of Experimental Psychology, University College London, 49 Gordon Square, London, United Kingdom, WC1H 0PD, or via e-mail: velia.cardin@gmail.com, velia.cardin@ucl.ac.uk.

Note

1. 

Compounds in BSL move from higher to lower locations, even in loans from English where the source has the reversed order, cf. “foot and mouth disease” in BSL is MOUTH FOOT DISEASE; “good night” is NIGHT GOOD, although “good morning” is GOOD MORNING, etc.

REFERENCES

REFERENCES
Agrafiotis
,
D.
,
Canagarajah
,
N.
,
Bull
,
D.
, &
Dye
,
M.
(
2003
).
Perceptually optimised sign language video coding based on eye tracking analysis
.
IEE Electronics Letters
,
39
,
1703
1705
.
Alivesatos
,
B.
, &
Petrides
,
M.
(
1997
).
Functional activation of the human brain during mental rotation
.
Neuropsychologia
,
35
,
111
118
.
Andin
,
J.
,
Orfanidou
,
E.
,
Cardin
,
V.
,
Holmer
,
E.
,
Capek
,
C. M.
,
Woll
,
B.
, et al
(
2013
).
Similar digit-based working memory in deaf signers and hearing nonsigners despite digit span differences
.
Frontiers in Psychology
,
4
,
942
.
Andin
,
J.
,
Rönnberg
,
J.
, &
Rudner
,
M.
(
2014
).
Deaf signers use phonology to do arithmetic
.
Learning and Individual Differences
,
32
,
246
253
.
Binder
,
J. R.
(
2012
).
Task-induced deactivation and the “resting” state
.
Neuroimage
,
62
,
1086
1091
.
Binder
,
J. R.
,
Desai
,
R. H.
,
Graves
,
W. W.
, &
Conant
,
L. L.
(
2009
).
Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies
.
Cerebral Cortex
,
19
,
2767
2796
.
Bracci
,
S.
,
Ietswaart
,
M.
,
Peelen
,
M. V.
, &
Cavina-Pratesi
,
C.
(
2010
).
Dissociable neural responses to hands and non-hand body parts in human left extrastriate visual cortex
.
Journal of Neurophysiology
,
103
,
3389
3397
.
Brentari
,
D.
(
2002
).
Modality differences in sign language phonology and morphophonemics
. In
R. P.
Meier
,
K.
Cormier
, &
D.
Quinto-Pozos
(Eds.),
Modality and structure in signed and spoken languages
(pp.
35
64
).
Cambridge
:
Cambridge University Press
.
Buchsbaum
,
B.
,
Pickell
,
B.
,
Love
,
T.
,
Hatrak
,
M.
,
Bellugi
,
U.
, &
Hickok
,
G.
(
2005
).
Neural substrates for verbal working memory in deaf signers: fMRI study and lesion case report
.
Brain and Language
,
95
,
265
272
.
Capek
,
C. M.
,
Macsweeney
,
M.
,
Woll
,
B.
,
Waters
,
D.
,
McGuire
,
P. K.
,
David
,
A. S.
, et al
(
2008
).
Cortical circuits for silent speechreading in deaf and hearing people
.
Neuropsychologia
,
46
,
1233
1241
.
Capek
,
C. M.
,
Waters
,
D.
,
Woll
,
B.
,
MacSweeney
,
M.
,
Brammer
,
M. J.
,
McGuire
,
P. K.
, et al
(
2008
).
Hand and mouth: Cortical correlates of lexical processing in British Sign Language and speechreading English
.
Journal of Cognitive Neuroscience
,
20
,
1220
1234
.
Cardin
,
V.
,
Orfanidou
,
E.
,
Rönnberg
,
J.
,
Capek
,
M.
,
Rudner
,
M.
, &
Woll
,
B.
(
2013
).
Dissociating cognitive and sensory neural plasticity in human superior temporal cortex
.
Nature Communications
,
4
,
1473
.
Carreiras
,
M.
,
Gutiérrez-Sigut
,
E.
,
Baquero
,
S.
, &
Corina
,
D.
(
2008
).
Lexical processing in Spanish Sign Language (LSE)
.
Journal of Memory and Language
,
58
,
100
122
.
Chen
,
Q.
,
Weidner
,
R.
,
Vossel
,
S.
,
Weiss
,
P. H.
, &
Fink
,
G. R.
(
2012
).
Neural mechanisms of attentional reorienting in three-dimensional space
.
Journal of Neuroscience
,
32
,
13352
13362
.
Cole
,
M. W.
, &
Schneider
,
W.
(
2007
).
The cognitive control network: Integrated cortical regions with dissociable functions
.
Neuroimage
,
37
,
343
360
.
Colin
,
C.
,
Zuinen
,
T.
,
Bayard
,
C.
, &
Leybaert
,
J.
(
2013
).
Phonological processing of rhyme in spoken language and location in sign language by deaf and hearing participants: A neurophysiological study
.
Neurophysiologie Clinique/Clinical Neurophysiology
,
43
,
151
160
.
Corina
,
D.
, &
Knapp
,
H.
(
2006
).
Sign language processing and the mirror neuron system
.
Cortex
,
42
,
529
539
.
Corina
,
D. P.
(
2000
).
Some observations regarding paraphasia and American Sign Language
. In
K.
Emmorey
&
H.
Lane
(Eds.),
The signs of language revisited: An anthology to honor Ursula Bellugi and Edward Klima
(pp.
493
507
).
Mahwah, NJ
:
Erlbaum
.
Corina
,
D. P.
,
Lawyer
,
L. A.
, &
Cates
,
D.
(
2012
).
Cross-linguistic differences in the neural representation of human language: Evidence from users of signed languages
.
Frontiers in Psychology
,
3
,
587
.
Corina
,
D. P.
,
McBurney
,
S. L.
,
Dodrill
,
C.
,
Hinshaw
,
K.
,
Brinkley
,
J.
, &
Ojemann
,
G.
(
1999
).
Functional roles of Broca's area and SMG: Evidence from cortical stimulation mapping in a deaf signer
.
Neuroimage
,
10
,
570
581
.
Costantini
,
M.
,
Galati
,
G.
,
Ferretti
,
A.
,
Caulo
,
M.
,
Tartaro
,
A.
,
Romani
,
G. L.
, et al
(
2005
).
Neural systems underlying observation of humanly impossible movements: An fMRI study
.
Cerebral Cortex
,
15
,
1761
1767
.
Craik
,
F. I. M.
, &
Lockhart
,
R. S.
(
1972
).
Levels of processing: A framework for memory research
.
Journal of Verbal Learning and Verbal Behavior
,
11
,
671
684
.
De Zubicaray
,
G. I.
,
Rose
,
S. E.
, &
McMahon
,
K. L.
(
2011
).
The structure and connectivity of semantic memory in the healthy older adult brain
.
Neuroimage
,
54
,
1488
1494
.
Devlin
,
J. T.
,
Matthews
,
P. M.
, &
Rushworth
,
M. F. S.
(
2003
).
Semantic processing in the left inferior prefrontal cortex: A combined functional magnetic resonance imaging and transcranial magnetic stimulation study
.
Journal of Cognitive Neuroscience
,
15
,
71
84
.
Dye
,
M. W. G.
, &
Shih
,
S.-I.
(
2006
).
Phonological priming in British Sign Language
. In
L.
Goldstein
,
D. H.
Whalen
, &
C. T.
Best
(Eds.),
Laboratory phonology
(pp.
243
263
).
Berlin
:
Mouton de Gruyter
.
Emmorey
,
K.
(
2002
).
Language, cognition, and the brain: Insights from sign language research
.
Mahwah, NJ
:
Lawrence Erlbaum and Associates
.
Emmorey
,
K.
,
Damasio
,
H.
,
McCullough
,
S.
,
Grabowski
,
T.
,
Ponto
,
L. L. B.
,
Hichwa
,
R. D.
, et al
(
2002
).
Neural systems underlying spatial language in American Sign Language
.
Neuroimage
,
17
,
812
824
.
Emmorey
,
K.
,
Grabowski
,
T.
,
McCullough
,
S.
,
Damasio
,
H.
,
Ponto
,
L. L. B.
,
Hichwa
,
R. D.
, et al
(
2003
).
Neural systems underlying lexical retrieval for sign language
.
Neuropsychologia
,
41
,
85
95
.
Emmorey
,
K.
,
McCullough
,
S.
, &
Brentari
,
D.
(
2003
).
Categorical perception in American Sign Language
.
Language and Cognitive Processes
,
18
,
21
45
.
Emmorey
,
K.
,
Mehta
,
S.
, &
Grabowski
,
T. J.
(
2007
).
The neural correlates of sign versus word production
.
Neuroimage
,
36
,
202
208
.
Emmorey
,
K.
,
Xu
,
J.
, &
Braun
,
A.
(
2011
).
Neural responses to meaningless pseudosigns: Evidence for sign-based phonetic processing in superior temporal cortex
.
Brain and Language
,
117
,
34
38
.
Felician
,
O.
,
Anton
,
J.-L.
,
Nazarian
,
B.
,
Roth
,
M.
,
Roll
,
J.-P.
, &
Romaiguère
,
P.
(
2009
).
Where is your shoulder? Neural correlates of localizing others' body parts
.
Neuropsychologia
,
47
,
1909
1916
.
Fine
,
I.
,
Finney
,
E. M.
,
Boynton
,
G. M.
, &
Dobkins
,
K. R.
(
2005
).
Comparing the effects of auditory deprivation and sign language within the auditory and visual cortex
.
Journal of Cognitive Neuroscience
,
17
,
1621
1637
.
Gentilucci
,
M.
, &
Dalla Volta
,
R.
(
2008
).
Spoken language and arm gestures are controlled by the same motor control system
.
Quarterly Journal of Experimental Psychology
,
61
,
944
957
.
Gilbert
,
S. J.
,
Bird
,
G.
,
Frith
,
C. D.
, &
Burgess
,
P. W.
(
2012
).
Does “task difficulty” explain “task-induced deactivation?”
.
Frontiers in Psychology
,
25
,
125
.
Grosvald
,
M.
,
Lachaud
,
C.
, &
Corina
,
D.
(
2012
).
Handshape monitoring: Evaluation of linguistic and perceptual factors in the processing of American Sign Language
.
Language and Cognitive Processes
,
27
,
117
141
.
Gutiérrez
,
E.
,
Müller
,
O.
,
Baus
,
C.
, &
Carreiras
,
M.
(
2012
).
Electrophysiological evidence for phonological priming in Spanish Sign Language lexical access
.
Neuropsychologia
,
50
,
1335
1346
.
Gutiérrez
,
E.
,
Williams
,
D.
,
Grosvald
,
M.
, &
Corina
,
D.
(
2012
).
Lexical access in American Sign Language: An ERP investigation of effects of semantics and phonology
.
Brain Research
,
1468
,
63
83
.
Hamilton
,
A. F.
, &
Grafton
,
S. T.
(
2009
).
Repetition suppression for performed hand gestures revealed by fMRI
.
Human Brain Mapping
,
30
,
2898
2906
.
Hartwigsen
,
G.
,
Baumgaertner
,
A.
,
Price
,
C. J.
,
Koehnke
,
M.
,
Ulmer
,
S.
, &
Siebner
,
H. R.
(
2010
).
Phonological decisions require both the left and right supramarginal gyri
.
Proceedings of the National Academy of Sciences, U.S.A.
,
107
,
16494
16499
.
Hedberg
,
T.
,
Almquist
,
S.
,
Ekevid
,
K.
,
Embacher
,
S.
,
Eriksson
,
L.
,
Johansson
,
L.
,
et al
(Eds.) (
2005
).
Svenskt Teckenspråkslexikon [Swedish Sign Language Dictionary]
.
Leksand, Sweden
:
Sveriges Dövas Riksförbund
.
Hickok
,
G.
, &
Poeppel
,
D.
(
2007
).
The cortical organization of speech processing
.
Nature Reviews Neuroscience
,
8
,
393
402
.
Husain
,
F. T.
,
Patkin
,
D. J.
,
Kim
,
J.
,
Braun
,
A. R.
, &
Horwitz
,
B.
(
2012
).
Dissociating neural correlates of meaningful emblems from meaningless gestures in deaf signers and hearing nonsigners
.
Brain Research
,
1478
,
24
35
.
Jordan
,
K.
,
Heinze
,
H. J.
,
Lutz
,
K.
,
Kanowski
,
M.
, &
Jancke
,
L.
(
2001
).
Cortical activations during the mental rotation of different visual objects
.
Neuroimage
,
13
,
143
152
.
Karnopp
,
L. B.
(
2002
).
Phonology acquisition in Brazilian Sign Language
. In
G.
Morgan
&
B.
Woll
(Eds.),
Directions in sign language acquisition
(pp.
29
53
).
Amsterdam
:
John Benjamins
.
Kelley
,
W. M.
,
Macrae
,
C. N.
,
Wyland
,
C. L.
,
Caglar
,
S.
,
Inati
,
S.
, &
Heatherton
,
T. F.
(
2002
).
Finding the self? An event-related fMRI study
.
Journal of Cognitive Neuroscience
,
14
,
785
794
.
Koshino
,
H.
,
Carpenter
,
P. A.
,
Keller
,
T. A.
, &
Just
,
M. A.
(
2005
).
Interactions between the dorsal and the ventral pathways in mental rotation: An fMRI study
.
Cognitive, Affective & Behavioral Neuroscience
,
5
,
54
66
.
Lomber
,
S. G.
,
Meredith
,
M. A.
, &
Kral
,
A.
(
2010
).
Cross-modal plasticity in specific auditory cortices underlies visual compensations in the deaf
.
Nature Neuroscience
,
13
,
1421
1427
.
MacSweeney
,
M.
,
Brammer
,
M. J.
,
Waters
,
D.
, &
Goswami
,
U.
(
2009
).
Enhanced activation of the left inferior frontal gyrus in deaf and dyslexic adults during rhyming
.
Brain
,
132
,
1928
1940
.
MacSweeney
,
M.
,
Campbell
,
R.
,
Woll
,
B.
,
Giampietro
,
V.
,
David
,
A. S.
,
McGuire
,
P. K.
, et al
(
2004
).
Dissociating linguistic and nonlinguistic gestural communication in the brain
.
Neuroimage
,
22
,
1605
1618
.
MacSweeney
,
M.
,
Capek
,
C. M.
,
Campbell
,
R.
, &
Woll
,
B.
(
2008
).
The signing brain: The neurobiology of sign language
.
Trends in Cognitive Science
,
12
,
432
440
.
MacSweeney
,
M.
,
Waters
,
D.
,
Brammer
,
M. J.
,
Woll
,
B.
, &
Goswami
,
U.
(
2008
).
Phonological processing in deaf signers and the impact of age of first language acquisition
.
Neuroimage
,
40
,
1369
1379
.
MacSweeney
,
M.
,
Woll
,
B.
,
Campbell
,
R.
,
Calvert
,
G. A.
,
McGuire
,
P. K.
,
David
,
A. S.
, et al
(
2002
).
Neural correlates of British Sign Language comprehension: Spatial processing demands of topographic language
.
Journal of Cognitive Neuroscience
,
14
,
1064
1075
.
MacSweeney
,
M.
,
Woll
,
B.
,
Campbell
,
R.
,
McGuire
,
P. K.
,
David
,
A. S.
,
Williams
,
S. C. R.
, et al
(
2002
).
Neural systems underlying British Sign Language and audio-visual English processing in native users
.
Brain
,
125
,
1583
1593
.
McDermott
,
K. B.
,
Petersen
,
S. E.
,
Watson
,
J. M.
, &
Ojemann
,
J. G.
(
2003
).
A procedure for identifying regions preferentially activated by attention to semantic and phonological relations using functional magnetic resonance imaging
.
Neuropsychologia
,
41
,
293
303
.
Milner
,
A. D.
, &
Goodale
,
M. A.
(
1993
).
Visual pathways to perception and action
.
Progress in Brain Research
,
95
,
317
337
.
Morgan
,
G.
,
Barrett-Jones
,
S.
, &
Stoneham
,
H.
(
2007
).
The first signs of language: Phonological development in British Sign Language
.
Applied Psycholinguistics
,
28
,
3
.
Moser
,
D.
,
Fridriksson
,
J.
,
Bonilha
,
L.
,
Healy
,
E. W.
,
Baylis
,
G.
, &
Rorden
,
C.
(
2009
).
Neural recruitment for the production of native and novel speech sounds
.
Dissertation Abstracts International, B: Sciences and Engineering
,
46
,
549
557
.
Neville
,
H. J.
,
Bavelier
,
D.
,
Corina
,
D.
,
Rauschecker
,
J.
,
Karni
,
A.
,
Lalwani
,
A.
, et al
(
1998
).
Cerebral organization for language in deaf and hearing subjects: Biological constraints and effects of experience
.
Proceedings of the National Academy of Sciences, U.S.A.
,
95
,
922
929
.
Northoff
,
G.
, &
Bermpohl
,
F.
(
2004
).
Cortical midline structures and the self
.
Trends in Cognitive Sciences
,
8
,
102
107
.
Op de Beeck
,
H. P.
,
Brants
,
M.
,
Baeck
,
A.
, &
Wagemans
,
J.
(
2010
).
Distributed subordinate specificity for bodies, faces, and buildings in human ventral visual cortex
.
Neuroimage
,
49
,
3414
3425
.
Orfanidou
,
E.
,
Adam
,
R.
,
McQueen
,
J. M.
, &
Morgan
,
G.
(
2009
).
Making sense of nonsense in British Sign Language (BSL): The contribution of different phonological parameters to sign recognition
.
Memory & Cognition
,
37
,
302
315
.
Orfanidou
,
E.
,
Adam
,
R.
,
Morgan
,
G.
, &
McQueen
,
J. M.
(
2010
).
Recognition of signed and spoken language: Different sensory inputs, the same segmentation procedure
.
Journal of Memory and Language
,
62
,
272
283
.
Pelphrey
,
K. A.
,
Morris
,
J. P.
,
Michelich
,
C. R.
,
Allison
,
T.
, &
McCarthy
,
G.
(
2005
).
Functional anatomy of biological motion perception in posterior temporal cortex: An fMRI study of eye, mouth and hand movements
.
Cerebral Cortex
,
15
,
1866
1876
.
Petitto
,
L. A.
,
Zatorre
,
R. J.
,
Gauna
,
K.
,
Nikelski
,
E. J.
,
Dostie
,
D.
, &
Evans
,
A. C.
(
2000
).
Speech-like cerebral activity in profoundly deaf people processing signed languages: Implications for the neural basis of human language
.
Proceedings of the National Academy of Sciences, U.S.A.
,
97
,
13961
13966
.
Price
,
C. J.
(
2012
).
A review and synthesis of the first 20 years of PET and fMRI studies of heard speech, spoken language and reading
.
Neuroimage
,
62
,
816
847
.
Price
,
C. J.
,
Moore
,
C. J.
,
Humphreys
,
G. W.
, &
Wise
,
R. J.
(
1997
).
Segregating semantic from phonological processes during reading
.
Journal of Cognitive Neuroscience
,
9
,
727
733
.
Puce
,
A.
,
Allison
,
T.
,
Bentin
,
S.
,
Gore
,
J. C.
, &
McCarthy
,
G.
(
1998
).
Temporal cortex activation in humans viewing eye and mouth movements
.
Journal of Neuroscience
,
18
,
2188
2199
.
Rudner
,
M.
,
Andin
,
J.
, &
Rönnberg
,
J.
(
2009
).
Working memory, deafness and sign language
.
Scandinavian Journal of Psychology
,
50
,
495
505
.
Rudner
,
M.
,
Karlsson
,
T.
,
Gunnarsson
,
J.
, &
Rönnberg
,
J.
(
2013
).
Levels of processing and language modality specificity in working memory
.
Neuropsychologia
,
51
,
656
666
.
Sandler
,
W.
, &
Lillo-Martin
,
D.
(
2006
).
Sign language and linguistic universals
.
Cambridge
:
Cambridge University Press
.
Siedlecki
,
T.
, &
Bonvillian
,
J. D.
(
1993
).
Location, handshape & movement: Young children's acquisition of the formational aspects of American Sign Language
.
Sign Language Studies
,
1078
,
31
52
.
Sliwinska
,
M. W.
,
Khadilkar
,
M.
,
Campbell-Ratcliffe
,
J.
,
Quevenco
,
F.
, &
Devlin
,
J. T.
(
2012
).
Early and sustained supramarginal gyrus contributions to phonological processing
.
Frontiers in Psychology
,
3
,
161
.
Söderfeldt
,
B.
,
Rönnberg
,
J.
, &
Risberg
,
J.
(
1994
).
Regional cerebral blood flow in sign language users
.
Brain and Language
,
46
,
59
68
.
Sutton-Spence
,
R.
, &
Woll
,
B.
(
1999
).
The linguistics of British Sign Language: An introduction
.
Cambridge
:
Cambridge University Press
.
Thompson
,
R. L.
,
Vinson
,
D. P.
, &
Vigliocco
,
G.
(
2010
).
The link between form and meaning in British Sign Language: Effects of iconicity for phonological decisions
.
Journal of Experimental Psychology
,
36
,
1017
1027
.
Ungerleider
,
L. G.
, &
Haxby
,
J. V.
(
1994
).
“What” and “where” in the human brain
.
Current Opinion in Neurobiology
,
4
,
157
165
.
Vingerhoets
,
G.
,
de Lange
,
F. P.
,
Vandemaele
,
P.
,
Deblaere
,
K.
, &
Achten
,
E.
(
2002
).
Motor imagery in mental rotation: An fMRI study
.
Neuroimage
,
17
,
1623
1633
.
Vinson
,
D. P.
,
Cormier
,
K.
,
Denmark
,
T.
,
Schembri
,
A.
, &
Vigliocco
,
G.
(
2008
).
The British Sign Language (BSL) norms for age of acquisition, familiarity, and iconicity
.
Behavior Research Methods
,
40
,
1079
1087
.
Visser
,
M.
,
Jefferies
,
E.
,
Embleton
,
K. V.
, &
Lambon Ralph
,
M. A.
(
2012
).
Both the middle temporal gyrus and the ventral anterior temporal area are crucial for multimodal semantic processing: Distortion-corrected fMRI evidence for a double gradient of information convergence in the temporal lobes
.
Journal of Cognitive Neuroscience
,
24
,
1766
1778
.
Willems
,
R. M.
, &
Hagoort
,
P.
(
2007
).
Neural evidence for the interplay between language, gesture, and action: A review
.
Brain and Language
,
101
,
278
289
.
Wilson
,
K. D.
, &
Farah
,
M. J.
(
2006
).
Distinct patterns of viewpoint-dependent BOLD activity during common-object recognition and mental rotation
.
Perception
,
35
,
1351
1366
.
Wilson
,
M.
, &
Emmorey
,
K.
(
1997
).
A visuospatial “phonological loop” in working memory: Evidence from American Sign Language
.
Memory & Cognition
,
25
,
313
320
.

Author notes

*

These authors contributed equally to this study.