Individuals who are deaf since early life may show enhanced performance at some visual tasks, including discrimination of directional motion. The neural substrates of such behavioral enhancements remain difficult to identify in humans, although neural plasticity has been shown for early deaf people in the auditory and association cortices, including the primary auditory cortex (PAC) and STS region, respectively. Here, we investigated whether neural responses in auditory and association cortices of early deaf individuals are reorganized to be sensitive to directional visual motion. To capture direction-selective responses, we recorded fMRI responses frequency-tagged to the 0.1-Hz presentation of central directional (100% coherent random dot) motion persisting for 2 sec contrasted with nondirectional (0% coherent) motion for 8 sec. We found direction-selective responses in the STS region in both deaf and hearing participants, but the extent of activation in the right STS region was 5.5 times larger for deaf participants. Minimal but significant direction-selective responses were also found in the PAC of deaf participants, both at the group level and in five of six individuals. In response to stimuli presented separately in the right and left visual fields, the relative activation across the right and left hemispheres was similar in both the PAC and STS region of deaf participants. Notably, the enhanced right-hemisphere activation could support the right visual field advantage reported previously in behavioral studies. Taken together, these results show that the reorganized auditory cortices of early deaf individuals are sensitive to directional motion. Speculatively, these results suggest that auditory and association regions can be remapped to support enhanced visual performance.
The absence of sensory inputs from one modality early in life has been linked to enhancement of the other senses. Accordingly, congenitally deaf people have been shown to display better performance at some visual tasks than hearing individuals (e.g., Shiell, Champoux, & Zatorre, 2014; Bottari, Nava, Ley, & Pavani, 2010; Dye, Hauser, & Bavelier, 2009; Lore & Song, 1991; Neville & Lawson, 1987; Parasnis & Samar, 1985). For instance, an enhancement at detecting and discriminating directional visual motion has been reported in early deaf people (Shiell et al., 2014; Hauthal, Sandmann, Debener, & Thorne, 2013; in the right visual field [RVF] only: Bosworth, Petrich, & Dobkins, 2013; Bosworth & Dobkins, 1999; Neville & Lawson, 1987). From an ecological perspective, the daily importance of visual motion may be increased for deaf individuals, especially for monitoring the peripheral visual field, for example, when using sign language (Codina, Pascalis, Baseler, Levine, & Buckley, 2017). However, for other potentially useful visual tasks, no differences or a decrease in performance has been reported across deaf and hearing people (for reviews on this controversy, see Pavani & Bottari, 2012; Mitchell & Maslin, 2007; Bavelier, Dye, & Hauser, 2006; Parasnis, 1983). The prevalent hypothesis explaining these differences regards neural plasticity, that is, the recruitment of brain areas processing the deprived sense or the reorganization of brain areas processing the existent senses or engaging in multisensory integration. It is thought that neural plasticity could support compensatory behavioral abilities, but only when the underlying functional organization of the incoming sense is compatible with those areas (e.g., Bola et al., 2017; Pascual-Leone & Hamilton, 2001). However, the capacity for neural plasticity of early deaf individuals to support behavioral advantages in visual tasks, including those involving motion, has not been clearly demonstrated.
Extensive neural plasticity has been reported for deaf individuals' responses to visual motion. Most strikingly, several human neuroimaging studies have reported activation in the primary auditory cortex (PAC) of deaf participants in response to moving or flickering visual stimuli, most often presented in or toward the visual periphery (peripheral moving dot pattern: Finney, Fine, & Dobkins, 2001; flickering patch of a full-field luminance grating: Finney, Clementz, Hickok, & Dobkins, 2003; peripheral moving dot pattern: Fine, Finney, Boynton, & Dobkins, 2005; flickering point lights in the RVF: Scott, Karns, Dow, Stevens, & Neville, 2014). In addition to auditory cortex, in the multisensory STS region (a term used to include the STS and adjacent cortex of the superior and middle temporal gyrus and angular gyrus; Allison, Puce, & McCarthy, 2000), a trend has been shown for higher activation and significantly more pronounced attentional enhancement in deaf people in response to visual dot motion (Bavelier et al., 2001). In the study by Scott et al. (2014), a larger area of activation around the STS was reported in deaf participants, including the posterior superior and middle temporal gyrus. Changes in responsiveness to peripherally presented visual motion or flickering stimuli have also been reported in human visual area hMT+: Increased (left-hemisphere) activation and/or extent of activation has been reported in deaf people (Scott et al., 2014; Bavelier et al., 2000, 2001; but see also Fine et al., 2005). To a lesser degree, other areas implying cross-modal neuroplasticity for motion or flicker in the early deaf people include the posterior parietal cortex, anterior cingulate, and FEF/supplementary eye field (Scott et al., 2014; Bavelier et al., 2001).
Again, however, the relationship between such neural plasticity in early deaf people and behavioral advantages in visual motion detection or discrimination has not been well documented. Recent evidence from animal studies suggests a causal link between reversible lesions in the auditory cortex and behavioral advantages at visual localization and movement detection in cats (Lomber, Meredith, & Kral, 2010; see also Meredith et al., 2011). Yet for humans, only noninvasive, correlative evidence has been provided. Structurally, for example, correlations have been found for deaf individuals between the relative amount of auditory cortex (planum temporale) or visual cortex (V1) devoted to processing peripheral motion and behavioral performance in motion detection tasks (auditory: Shiell, Champoux, & Zatorre, 2016; visual: Levine, Codina, Buckley, de Sousa, & Baseler, 2015). Suggestive evidence has also been provided by showing that the recruitment of reorganized brain regions in early deaf individuals shows selective responses to a visual task for which there is behavioral enhancement. For example, four cardinal locations of visual stimuli could be decoded from the auditory cortex in deaf individuals with neuroimaging, suggesting that representations in the auditory cortex align with those in the visual cortex (Almeida et al., 2015). Here, we aim to add to these findings by asking whether deaf individuals' enhanced ability in speed and/or accuracy at discriminating the direction of visual motion could be supported by direction-selective responses in brain areas evidencing neural plasticity.
Directional visual motion is a particularly salient visual stimulus and is known to selectively activate a subset of areas in the neurotypical human brain responding to visual motion more generally. Strong direction-selective responses have been found in human visual area hMT+/V5 (e.g., Huk, Ress, & Heeger, 2001; Morrone et al., 2000; Tootell et al., 1995). Other implicated areas include V3/V3A and, to a lesser extent, the rest of V1–V4 (Ales & Norcia, 2009; Huk et al., 2001; Tootell et al., 1995; finding large effects also in V1 with EEG source imaging). The representation of directional motion within these cortical areas was first revealed by single-cell recordings in monkeys, reporting columnar direction tuning (Felleman & Van Essen, 1987; Albright, 1984; Dubner & Zeki, 1971; Hubel & Wiesel, 1961). Unfortunately, because of the spatial scale, such direction tuning cannot be studied noninvasively in humans, and direction-specific representation in humans has thus remained elusive (see Kamitani & Tong, 2006, for a potential exception, but also Beckett, Peirce, Sanchez-Panchuelo, Francis, & Schluppeck, 2012; for axis of motion mapping at 7 T, see Zimmermann et al., 2011). Despite this, direction-selective areas may be identified in the human brain with fMRI with stimulation presentation techniques, such as contrasting directional (i.e., coherent) motion with directionless (i.e., noncoherent) motion or dynamic noise (Morrone et al., 2000; Beauchamp, Cox, & DeYoe, 1997; Braddick, Hartley, Atkinson, Wattam-Bell, & Turner, 1997; in EEG/magnetoencephalography: Palomares, Ales, Wade, Cottereau, & Norcia, 2012; Ales & Norcia, 2009; Nakamura et al., 2003; Lam et al., 2000; Tyler & Kaitz, 1977).
Here, we used a sensitive approach to investigate the spatial extent and activation of direction-selective brain regions in early deaf and hearing people, focusing on the auditory and association cortices, the PAC and STS region, respectively, in comparison with visual area hMT+. Specifically, we used fMRI together with a frequency-tagging approach (e.g., Gao, Gentile, & Rossion, 2017; Koening-Robert, VanRullen, & Tsuchiya, 2015; Ernst, Boynton, & Jazayeri, 2013; Morrone et al., 2000; Engel, Zhang, & Wandell, 1997; Puce, Allison, Gore, & McCarthy, 1995; Bandettini, Jesmanowicz, Wong, & Hyde, 1993) to identify periodic changes from noncoherent to directional random-dot motion (Morrone et al., 2000; see also Palomares et al., 2012; Ales & Norcia, 2009; Atkinson et al., 2008). We were thus able to acquire signals with a high signal-to-noise ratio that were independent of a hemodynamic response function model. By using a contrast of directionless-to-directional motion, we were also able to capture direction-selective responses (note that these responses are not direction specific) locked precisely to the frequency of coherence onset. To follow up on a behavioral advantage for direction discrimination typically reported in the RVF for deaf individuals (Bosworth et al., 2013; Bosworth & Dobkins, 1999; Neville & Lawson, 1987), we presented visual stimuli in the left visual field (LVF) and RVF as well as centrally. When activation was found, we further explored potential qualitative differences across hearing and deaf individuals in terms of spatial extent, RVF versus LVF response, and hemispheric lateralization. Together, these comparisons allowed us to assess the potential neural bases of enhanced visual motion processing reported in previous studies for early deaf people.
Two groups of participants, early deaf and hearing controls, were tested in the experiment, which was approved by the institution review board of the University of Nevada, Reno, and conducted in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki). Each group consisted of six adults, recruited from northern Nevada and California. Our deaf participants included those who experienced severe-to-profound sensorineural hearing loss at an early age. They had no ability to understand auditory speech but were proficient in sign language (see Table 1 for deaf participants' details). The mean age of deaf participants was 36 years (SD = 8.2, range = 26–49 years); the mean age of hearing participants was 33 years (SD = 8.5, range = 26–48 years). Four of the hearing and one of the deaf participants were male; one hearing participant was left-handed. All participants were unaware of the experimental design, except for one hearing participant, who was author T. L. R. All participants reported visual acuity in the normal or corrected-to-normal range.
|.||Age (Years) .||Sex .||Handedness .||Deafness Acquisition .||Cause of Deafness .||Auditory Deprivation, Left/Right (dB) .||Signing Acquisition .|
|D1||41||F||R||12 months||Unknown||95/95||12 years|
|D2||31||M||R||15 months||Fever||Total/85||15 months|
|D3||49||F||R||Birth||Maternal gestational measles||100/90||11 years|
|D4||26||F||R||Birth||Genetic (coex26)||85/85||< 1 year|
|D5||34||F||R||Birth||Hereditary||80/70||< 1 year|
|.||Age (Years) .||Sex .||Handedness .||Deafness Acquisition .||Cause of Deafness .||Auditory Deprivation, Left/Right (dB) .||Signing Acquisition .|
|D1||41||F||R||12 months||Unknown||95/95||12 years|
|D2||31||M||R||15 months||Fever||Total/85||15 months|
|D3||49||F||R||Birth||Maternal gestational measles||100/90||11 years|
|D4||26||F||R||Birth||Genetic (coex26)||85/85||< 1 year|
|D5||34||F||R||Birth||Hereditary||80/70||< 1 year|
F = female; M = male; R = right.
fMRI scanning was performed with a 3-T Philips Ingenia scanner using a 32-channel digital SENSE head coil (Philips Medical Systems) at the Renown Regional Medical Center, Reno, NV. Volumetric anatomical images were acquired at a resolution of 1 mm3 using a T1-weighted magnetization prepared rapid gradient echo sequence. Functional BOLD signals were acquired through a continuous design at a resolution of 2.75 × 2.75 × 3 mm voxels, with no gap. A repetition time of 2 sec was used to acquire 30 transverse slices in an ascending order, with an echo time of 17 msec, a flip angle of 76°, and a 220 × 220 mm2 field of view.
Visual Motion Stimuli
Visual motion was displayed with random-dot kinematograms, based on the incremental displacement across monitor refresh frames of individual dots within a circular field (Braddick, 1974; Julesz, 1971; Anstis, 1970). Frames of white dots against a black background were generated with a custom script running over MATLAB (The MathWorks), refreshing at a rate of about 60 Hz, with a 500-msec lifespan to discourage participants tracking the movement of individual dots. Given some inconsistency in presentation timing because of online drawing of dot positions, the motion display was adapted for precise periodic stimulus presentation by exporting generated dot motion frames and then displaying them at a precisely controlled periodic rate of 60 Hz using custom software running over Java. Viewed on the testing monitor, the stimulus field diameter subtended 8.5° of visual angle, with individual dots subtending 1.35 in. in diameter, moving at a speed of 3.4°/sec, at a density of 12.5 dots/deg.
We created directional stimulus sequences in four directions (up, right, down, and left) as well as nondirectional, noncoherent dot motion. To create the directional sequences, a 30-sec sequence of 1,800 sequential stimulus frames creating the appearance of 100% coherent rightward visual motion was extracted. The rightward stimuli frames were rotated by increments of 90° to create sequences of downward, leftward, and upward motions, respectively, with minimal variation across directions. In the functional scans, these sequences each repeated four times in immediate succession, leading to a block of 2 sec of directional motion. To create sequences of nondirectional motion, 100% noncoherent motion stimulus frames were similarly extracted from 30-sec sequences; to fill the longer proportion of nondirectional-to-directional motion duration in the testing sequences with consistent stimulus update intervals, this procedure was repeated three additional times. Note that these 30-sec sequence pieces also served as “incoherent jumps” to prevent a specific confound of full dot replacement at the onset and offset times of directional and nondirectional motion (see the following section; Braddick, Birtles, Wattam-Bell, & Atkinson, 2005; Wattam-Bell, 1991). In functional scans, these four nondirectional sequence sets were each repeated four times in immediate succession, defining a block of 8 sec of nondirectional motion. Participants viewed the stimulation monitor with a mirror attached to the MR head coil.
Periodic Visual Stimulation Procedure
Functional scans consisted of periodic alternation between directional and nondirectional motion over a duration of 5.1 min. Scans began with 2 sec of a white fixation cross on a black background, followed by a 2-sec fade-in period, in which stimulus luminance contrast gradually increased to 100%. Stimuli were then shown over a duration of 300 sec in a fixed pattern of 2 sec of directional motion followed by 8 sec of nondirectional motion. Periods of directional motion thus onset every 10 sec, leading to a direction-selective frequency-tagged rate of 1/10 sec, that is, 0.1 Hz (Figure 1A). Within each scan, the direction of motion also consistently alternated at each presentation cycle, for example, from upward to downward motion, leading to a direction-specific frequency-tagged rate of 0.05 Hz. Finally, the scans ended with 2 sec of stimulus fade-out and 2 sec of the white fixation cross. Four participants from each of the deaf and hearing groups saw contrasts of up/down and left/right motions (Trial Lists 1 and 2), and the remaining two participants of each group saw contrasts of up/left and right/down motions (Trial Lists 3 and 4). Because no clusters of significant responses to direction-specific motion at 0.05 Hz were found for any participants in any trial lists, data were combined across trial lists within each group to examine the direction-selective response at 0.1 Hz.
Visual Field Conditions
Scans designed to localize brain regions responding to visual motion contained stimuli presented in a central visual field (CVF) condition. In two additional scan conditions designed to measure the amplitude of brain activation, stimuli were presented in either the right or left peripheral visual field. In the CVF condition, stimuli were presented in the center of the stimulation monitor together with a superimposed central fixation cross. From a viewing distance of 134 cm, the monitor supported a field of view of 29° × 17°. Thus, when presented in the right or left peripheral visual field conditions, the stimulus was translated laterally to the edge of the monitor and the fixation cross shifted laterally 4° from center in the opposite direction, so that the distance between the proximal edge of the stimulus and the fixation cross subtended 10° (e.g., Jiang, Beauchamp, & Fine, 2015). Four scan repetitions of each condition were presented sequentially to discourage participants from moving their heads as stimulus location changed. Each participant was presented with every condition, leading to 12 scans for a total testing time of about 1 hr. In odd trial lists, scans began with stimuli presented in the CVF, whereas in even trial lists, stimuli were first presented in one of the peripheral visual fields. The order of conditions was identical across participant groups.
Participants were instructed to fixate on the centrally presented white fixation cross. The cross changed shape to a circle for a duration of 200 msec at random intervals (a minimum of 800 msec in between changes) 30 times within each scan, that is, once about every 10 sec. Participants were asked to use a response box to report the direction of motion at the time of the fixation shape change. This task was designed to facilitate participants' fixation as well as to encourage attention to the direction of motion of the stimulus.
fMRI Data Analysis
Anatomical and functional data were analyzed with BrainVoyager v20.0 and the BVQXTools toolbox (Brain Innovation B.V.) together with MATLAB R2013b. Functional scan data were imported into BrainVoyager and preprocessed with corrections for slice scan time and 3-D motion (aligned to the first functional scan for intersession alignment). They were temporally filtered with a linear de-trending; no spatial smoothing was applied. Anatomical scans, similarly imported into BrainVoyager, were subjected to an isotropic voxel transformation and aligned according to standard anterior and posterior commissure points. For display across participants, data were transformed into a conventional Talairach space (Talairach & Tournoux, 1988). Functional scans were coregistered to each participant's corresponding anatomical images. Initial alignment was fine-tuned through an affine transformation and minimally corrected with visual inspection. Spatial normalization of the functional data was applied through a volume time course transformation.
Frequency Domain Processing
The volume time course files of each functional scan were imported into MATLAB for frequency domain analyses. They were cropped to 150 volumes of 2 sec, containing exactly 30 presentation cycles of 0.1-Hz directional motion and excluding the first and last two volumes corresponding to fixation cross and fade-in/out presentation. BOLD data from each participant from the four scans per condition were averaged in time to reduce noise from non-phase-locked activation, that is, from activation not driven by periodic stimulus presentation. A DC correction was applied to remove the mean signal offset, and the data were transformed into a normalized amplitude spectrum through a fast Fourier transform (Figure 1B). The resulting BOLD amplitude spectrum contained a range of 0–0.25 Hz with a frequency resolution of 0.0033 Hz. For each frequency bin, x, a baseline range was defined as 20 surrounding frequency bins, encompassing a range of about 0.07 Hz centered around x. To assess significance during CVF scans of the 0.1-Hz response at each voxel, z scores were generated by subtracting from x the mean baseline value and dividing the result by the standard deviation of the baseline. To display BOLD response amplitudes in predetermined regions (see section below) during RVF and LVF scans, baseline-subtracted amplitude values were similarly generated by subtracting the mean baseline value from x (e.g., Retter & Rossion, 2016). The resulting files were reimported into BrainVoyager for display.
Given previously reported findings of neural plasticity in the PAC and association auditory cortex in deaf individuals (e.g., Scott et al., 2014; Karns, Dow, & Neville, 2012; Fine et al., 2005; Finney et al., 2001, 2003) and direction-selective responses in visual area hMT+ (e.g., Huk et al., 2001; Morrone et al., 2000; Tootell et al., 1995), we a priori focused our analyses on the PAC and STS region, potentially including part of the STS/PT, middle temporal gyrus, and angular gyrus (Allison et al., 2000; see also Scott et al., 2014, for activation in deaf participants), and hMT+.
To define the STS region and hMT+, we used a functional cluster-based criterion from direction-selective responses at 0.1 Hz to motion presented in the CVF (clusters > 150 voxels). Significance thresholding was applied at the individual participant level (range: z > 2.6 to z > 5.7), to approximately equalize the number of significant voxels across commonly active regions, including hMT+ (six deaf and six hearing, in at least one hemisphere), the STS region (six deaf and six hearing), early visual areas (six deaf and six hearing), and the lateral occipital complex (five deaf and six hearing). In relevant cases, the threshold was increased for two regions, applied bilaterally, to spatially separate them. The mean total voxel number across participants after thresholding was 15,138 voxels and did not differ significantly across groups (deaf: M = 13,636, SE = 1,422; hearing: M = 16,641, SE = 1,583), t = 1.41, p = .19, d = 0.73 (two-tailed).
In a separate analysis, we defined the PAC, a region that cannot be functionally defined in deaf participants, using the Julich probabilistic atlas in the SPM Anatomy Toolbox (Eickhoff, Heim, Zeilles, & Amunts, 2006; Eickhoff et al., 2005). Following a procedure described in Eickhoff et al. (2006), we included the volume assignment to all subregions for PAC (Morosan et al., 2001) in the summary map of all areas (maximum probability map). This procedure ensured no overlap between any two cytoarchitectonic defined areas. The PAC ROI was then transformed to Talairach space and applied to each participant's brain volume. It was further separated into left and right hemisphere PAC for each participant.
For the functionally defined ROIs, namely, the STS region and hMT+, we investigated whether there were significant differences in the spatial extent and amplitude of activation between the deaf and hearing participant groups. The spatial extent and amplitude of the STS region and hMT+ were thus compared across the deaf and hearing participant groups with nonparametric Mann–Whitney U tests, given the relatively small sample size. To compare differences in the spatial extent of the STS region and hMT+, the number of significant voxels was used. To compare the amplitude differences in these ROIs to stimuli presented in the LVF and RVF, baseline-subtracted amplitude values at 0.1 Hz for each participant were averaged across voxels within their individually defined ROIs for the LVF and RVF responses separately. When a cluster-based ROI could not have been defined in one hemisphere (STS: two deaf and one hearing in the left hemisphere; MT: one deaf participant in the left hemisphere), no corresponding amplitude values were included in the analysis.
For the probabilistically defined ROI, that is, the PAC, we investigated whether there were significant responses in the deaf and/or hearing participants. To determine response significance in the PAC ROI, an amplitude spectrum was computed from the averaged BOLD responses to motion presented in the CVF of all bilateral PAC voxels across participants in each group. z Scores were then calculated on this averaged spectrum with a threshold of p < .001 (z > 3.10) for significance for this sensitive group-level analysis. Given some debate about whether PAC responses occur only as a result of group level averaging (e.g., as shown in Finney et al., 2001; but see Scott et al., 2014), significance was also assessed similarly at the individual participant level, with the typical threshold of p < .05 (z > 1.64). To compare the number of significantly direction-selective voxels across the PAC and the STS region, the PAC ROI was thresholded at the individual level defined previously for demarcating the STS region (i.e., encompassing a range of z > 2.6 to z > 5.7).
The STS Region and Visual Middle Temporal Complex
The centrally presented visual motion trials were used to localize direction-selective responses in deaf and hearing individuals. These responses were frequency-tagged at 0.1 Hz, that is, the rate at which directional (100% dot coherence) motion onset (and continued for 2 sec) immediately after 8 sec of directionless motion (0% dot coherence).
Direction-selective Responses Are More Extensive in the Right STS Region for Deaf Participants
The area of the STS region was 5.5 times larger in deaf than hearing individuals in the right hemisphere (deaf: M = 3,591 mm3, SE = 596.2; hearing: M = 653 mm3, SE = 22.6), with no pronounced differences in the left hemisphere (deaf: M = 294 mm3, SE = 164.8; hearing: M = 315 mm3, SE = 124.8; Figure 2A). Statistically, this led to a significant difference in the extent of STS region activation across participant groups in the right STS region only: U = 1, p = .004 (left STS: U = 36, p = .70).
The STS region in the right hemisphere was centered at Talairach x = 54, y = −42, and z = 9 for deaf participants and x = 55, y = −39, and z = 16 for hearing participants (for individual regions, see Figure 3). The location of the STS region was particularly reliable in the right hemisphere for deaf participants; The range of its center Talairach x coordinates (see Figure 3) was x = 52–58 (SE = 0.84) for deaf participants, compared with x = 48–66 (SE = 3.07) for hearing participants (in the left hemisphere, the range was x = 45–61 [SE = 3.45] for deaf participants and x = 50–65 [SE = 2.96] for hearing participants).
The area of hMT+ did not appear to differ greatly across participant groups, although the right hemisphere (deaf: M = 1,847 mm3, SE = 183.3; hearing: M = 1,304 mm3, SE = 321.2) appeared larger than the left (deaf: M = 1,033 mm3, SE = 365.9; hearing: M = 1,335 mm3, SE = 291.1) for deaf participants only (Figure 2B). However, statistically, there was not a significant difference in the extent of hMT+ activation across deaf and hearing participants, in either the right, U = 11, p = .31, or left, U = 14, p = .59, hemisphere. In summary, the only significant difference found between deaf and hearing participants in terms of area of activation was a greater extent of the right STS region for deaf participants.
Responses in the LVF vs. RVF: An RVF STS Region Bias for Deaf Participants
The amplitude of responses within the STS region and hMT+ ROIs identified previously was used to quantify the response to visual motion presented in separate trials in the LVF and RVF. We expected that the larger size of the right STS region only in deaf individuals might be accompanied by enhanced activity in response to visual stimuli presented in the LVF. Instead, the results showed the opposite, that is, that STS region responses of deaf individuals were of larger amplitude for stimuli presented in the RVF than LVF (Figure 4A). Indeed, for stimuli presented in the RVF, there was a significantly higher response for deaf than hearing participants in the right STS, U = 0, p = .002, which only neared significance in the left STS, U = 2, p = .063. In contrast, for stimuli presented in the LVF, there were no significant differences across participant groups—right STS: U = 9, p = .18; left STS: U = 9, p = .91.
In hMT+, the pattern of amplitude responses to stimuli in the LVF and RVF appeared highly similar across the left and right hemispheres for deaf and hearing participants (Figure 4B). This pattern was described by a contralateral visual field to hemisphere advantage, particularly for the RVF/left hemisphere (see Figure 4B). Across participant groups, however, there were no significant differences for stimuli presented in either the RVF (right hMT+: U = 12, p = .39; left hMT+: U = 14, p = .93) or LVF (right hMT+: U = 15, p = .70; left hMT+: U = 15, p = 1.0). Overall, there were thus no differences between deaf and hearing participants in hMT+ responses to directional motion in the LVF or RVF, but deaf participants had more activation than hearing participants in the right STS region to stimuli presented in the RVF.
The PAC was defined with a probabilistic atlas for both deaf and hearing participants. To determine response significance in this region, an amplitude spectrum was derived from the averaged BOLD responses to motion presented in the CVF of all PAC voxels across participants for each group.
Significant But Minimal Direction-selective Responses in the PAC for Deaf Participants
At the individual participant level, direction selectivity at 0.1 Hz was evidenced in five of six deaf participants in the bilateral PAC (zs ranging from 1.79 to 3.19, ps < .05). A significant direction-selective response emerged at 0.1 Hz for the deaf participants across the bilateral auditory cortex (z = 5.80, p < .0001; right PAC: z = 7.78; left PAC: z = 3.40; Figure 5A).
A direction-selective response was not found in five of six hearing participants (all zs < 1.34, ps > .05); however, a significant response was found in one hearing participant who was not naive to the experimental design (z = 4.36, p < .0001). When including all six participants of the hearing group, the direction-selective response in the bilateral PAC reached significance at a threshold of p < .05 (bilaterally: z = 2.32, p = .010; right PAC: z = 1.88; left PAC: z = 2.27); when removing the nonnaive participant, the PAC was not significant in the hearing group (bilaterally: z = 1.39, p = .082).
Despite significant responses in the PAC in deaf participants, the extent of direction-selective responses was minimal, with significant voxels subtending only 14.1% of the bilateral PAC area (at z > 3.10, p < .001) for deaf participants at the group level (i.e., grand-averaged amplitude spectra; see Methods). When the lowest and highest z score thresholds applied for individual participants (z > 2.6 to z > 5.7) were applied to the group level data, the percentage of significant bilateral PAC area at the group level ranged from, respectively, 21.8% to 0.67%. To put this in perspective, this area was more than 28 times smaller than the extent of activation in the bilateral STS region for deaf participants when defined at the same significance threshold for individual participants (see Figure 5B; mean PAC area: 1.11%, SE = 0.43%). In summary, significant PAC responses were present but minimal in the deaf participant group.
Responses in the LVF vs. RVF: The PAC Hemispheric Activation Mirrors the STS
The responsivity of the BOLD amplitude of direction-selective responses to visual motion presented in the RVF or LVF was investigated, as in the Responses in the LVF vs. RVF: An RVF STS Region Bias for Deaf Participants section (Figure 5C). The resultant pattern of activation across hemispheres in the PAC was reminiscent of that of the STS region (compare Figure 5C with Figure 4A). Note that the large amplitude differences across the STS region and PAC are not comparable directly, because of the different methods of definition of these regions (i.e., the STS region was defined functionally to include only significant voxels, whereas the PAC was defined as all voxels within a predefined region).
We used an fMRI frequency-tagging approach to identify direction-selective brain regions in early deaf and hearing people, investigating the spatial extent of their activation (in response to stimuli presented in the CVF) and the amplitude of their activation (in response to stimuli presented separately in the LVF and RVF). We focused our analysis on the PAC and associative STS region, in comparison with visual area hMT+. We predicted that direction-selective response would be found in the PAC and STS region, in line with enhanced behavioral abilities reported for early deaf individuals in discriminating and/or detecting directional visual motion (Shiell et al., 2014; Bosworth et al., 2013; Hauthal et al., 2013; Bosworth & Dobkins, 1999; Neville & Lawson, 1987). Note that we are able to identify direction-selective responses emerging from a contrast of directional versus nondirectional visual motion in our frequency-tagging paradigm. Direction-selective motion responses are more selective than motion-selective responses but less selective than direction-specific (e.g., leftward-selective) responses. On the other hand, previous studies investigating motion-related responses in the early deaf people have reported motion-selective, rather than direction-selective, responses (e.g., Fine et al., 2005; Finney et al., 2001).
Direction-selective Responses Are Found in the STS Region for Both Hearing and Deaf Individuals
To our knowledge, this is the first study showing direction selectivity for translational visual motion in the human STS region (see Figures 2A and 3), here encompassing the posterior to middle STS, superior temporal gyrus, and middle temporal gyrus (for direction selectivity with rotational head and ellipsoid motion, see Carlin, Rowe, Kriegeskorte, Thompson, & Calder, 2012). The STS region is known to respond to visual (biological) motion in neurotypical humans and nonhuman animal models (for a review, see Allison et al., 2000; see also, e.g., Noguchi, Kaneoke, Kakigi, Tanabe, & Sadato, 2005; Grossman & Blake, 2001). Moreover, direction-selective tuning of single neurons to visual motion has been reported in the STS region of monkeys (e.g., Nelissen, Vanduffel, & Orban, 2006; Oram, Perrett, & Hietanen, 1993; Bruce, Desimone, & Gross, 1981; Zeki, 1978). The absence of direction-selective STS responses in past human neuroimaging or source localization studies may be for several reasons: For example, these studies focused on more traditionally, retinotopically defined areas, and there may be differences in activation resulting from the directional/nondirectional motion contrast used here and the motion adaptation paradigms favored previously. Note that, in previous studies, direction-selective responses were reported only in visual areas V1 through hMT+/V5 and the lateral occipital complex (Hong, Tong, & Seiffert, 2013; Ales & Norcia, 2009; Huk et al., 2001; Tootell et al., 1995). In addition, the frequency-tagging paradigm applied here may have provided methodological advantages, enabling a powerful contrast of directional and nondirectional motion, an analysis with a high signal-to-noise ratio, and not relying on a hemodynamic response function model (e.g., Gao et al., 2017; Koening-Robert et al., 2015; Ernst et al., 2013; Morrone et al., 2000; Engel et al., 1997; Puce et al., 1995; Bandettini et al., 1993).
The direction-selective STS region could be functionally defined in all individual deaf and hearing participants in the right hemisphere and in five deaf and four hearing participants in the left hemisphere (see Figure 3). It was 2–12 times larger in the right than left hemisphere, for the hearing and deaf participants, respectively (see Figure 2A). At the group level, in the right hemisphere, this region was centered at Talairach coordinates of x = 55, y = −39, and z = 16 for hearing participants and x = 54, y = −42, and z = 9 for deaf participants. The localization of the STS region here is similar to that reported in previous studies (e.g., for deaf participants, response to visual motion: x = 56, y = −40, z = 8, in Table 5 of Bavelier et al., 2001; for neurotypical participants in response to visual, tactile, and auditory stimuli: left anterior inferior coordinates of x = 52, y = 44, z = 15, in Beauchamp, Yasar, Frye, & Ro, 2008).
This STS region also showed a right-hemisphere advantage in terms of response amplitude to stimuli shown in the LVF and RVF, particularly for deaf participants. In contrast, there was no left-hemisphere advantage apparent for stimuli shown in the RVF for either participant group (see Figure 4A). These results are in line with larger responses to visual motion in the right hemisphere generally (e.g., Corballis, 2003; Finney et al., 2001; Kubova, Kuba, Hubacek, & Vit, 1990; see also Weeks et al., 2000, for an example of right-hemisphere dominance to auditory motion in congenitally blind participants) as well as interhemispheric transfer of visual motion information (Brandt, Stephan, Bense, Yousry, & Dieterich, 2000; see also Motter, Steinmetz, Duffy, & Mountcastle, 1987) and previous reports of no contralateral organization in the STS region (e.g., Grossman & Blake, 2001; see also Saygin & Sereno, 2008).
The PAC Shows Direction-selective Visual Motion Responses in Early Deaf Individuals
We discovered significant direction-selective responses to visual motion in a probabilistically defined PAC region in early deaf people (see Figure 5). The extent of this activation was highly dependent on the significance threshold used; at p < .001, it appeared to cover 14.1% of the bilateral PAC for the early deaf group. In comparison with the extent of activation in the STS at the same significance threshold, this area is more than 28 times smaller. Nevertheless, when averaging across all voxels in the bilateral PAC, a significant response emerged for five of six deaf participants (p < .05).
Responses to visual stimuli were first reported in the PAC for early deaf people in response to peripheral moving dots at the group level (Finney et al., 2001). PAC activation was replicated in the early deaf people in response to moving or flickering stimuli, most often in or near the visual periphery (Finney et al., 2001, 2003). Importantly, these results were likely not an effect of group averaging or imprecise PAC definition: A recent study identified PAC activation defined anatomically at the individual participant level, using the transverse temporal gyrus, also known as Heschl's gyrus, with flickering point lights in the RVF (Scott et al., 2014). In this study, the amount of activation in the PAC was reported only in comparison for peripherally versus perifoveally presented flicker dots, preventing a direct comparison with the extent or amount of activation reported here. Still, our finding that PAC activation is—at least to some extent—direction selective adds to our knowledge of neural plasticity in this region for early deaf people.
The pattern of PAC activation in response to stimuli presented in the RVF and LVF is highly reminiscent of that of the STS region (see Figures 5C and 4A). One possibility is that the PAC projects information into the STS region, a sensory association area (e.g., Hackett et al., 2007; Smiley et al., 2007; Seltzer et al., 1996; Seltzer & Pandya, 1978; Benevento, Fallon, Davis, & Rezak, 1977; see also Beauchamp et al., 2008). In addition, the STS also projects information back to the superior temporal gyrus (e.g., Barnes & Pandya, 1992, using retrograde tracing in the rhesus monkey), suggesting reciprocal connections and more complex interactions between these regions. Note that the correspondence between the PAC and STS region found here cannot be explained by overlap between these areas: There was no overlap in five deaf participants (0.8% for the remaining one participant) and no overlap at the group level in the right hemisphere.
The Right STS Region Is Recruited Extensively for Processing Direction-selective Visual Motion in Early Deaf Individuals
The most striking difference between deaf and hearing individuals in response to directional motion was found in the right STS region, which was 5.5 times larger for deaf than hearing participants (for a 12 times greater extent in the right posterior STS in deaf than hearing participants in response to attended visual motion, see Table 2 of Bavelier et al., 2001). In contrast, no differences in direction-selective responses were found across groups in the left STS region or visual area hMT+ here.
The STS is a likely region for cross-modal organization, as it covers an expansive region of the temporal lobe and expresses great functional diversity, containing subregions sensitive to auditory, visual, tactile, and multisensory stimuli (e.g., Dahl, Logothesis, & Kayser, 2009; Beauchamp et al., 2008; Beauchamp, Argall, Bodurka, Duyn, & Martin, 2004; Calvert, Campbell, & Brammer, 2000; Seltzer & Pandya, 1978; Benevento et al., 1977). The posterior STS receives inputs from both the visual and auditory cortex, whereas the middle STS normally receives auditory inputs only, at least in the rhesus monkey (Seltzer & Pandya, 1994); in humans, auditory–visual responses have been reported to be largest in the middle STS (Venezia et al., 2017). Congruently, the auditory association cortices have also been invoked in studies in neurotypical individuals on cross-modal plasticity through learned associations (e.g., Meyer, Baumann, Marchina, & Jancke, 2007; see also Bulkin & Groh, 2006; Ghazanfar & Schroeder, 2006).
In congenitally deaf people, greater connectivity between the middle STS across hemispheres, as well as with the ipsilateral posterior STS, hints at a reorganization of this region in line with cross-modal plasticity (Li, 2013). Specific examples of cross-modal plasticity in the STS region have been reported with regard to how early deaf people process sign language. Early deaf participants have been shown to have increased activation in the middle STS in response to sign language (e.g., Sadato et al., 2004; Neville et al., 1998). In addition, increased posterior STS activation was shown in deaf signers, and not hearing signers, when performing a velocity task (see Figure 6 of Bavelier et al., 2001). Our report of expansive recruitment of the STS region in early deaf people in response to visual motion thus further confirms a general pattern of neural plasticity in this region.
In hearing individuals, some authors claim that responses to auditory motion are separate from those to auditory localization and rely on the superior temporal gyrus (e.g., Ducommun et al., 2002, 2004; Baumgart, Gaschler-Markefski, Woldorff, Heinze, & Scheich, 1999). However, others claim that the auditory cortex may be selective to spatial locations rather than motion (e.g., Smith, Okada, Saberi, & Hickok, 2004). Our findings suggest that, at least in response to visual motion, auditory areas in deaf participants and association areas in deaf and hearing participants are selectively responsive to directional visual motion.
Direction-selective Responses to Stimuli in the RVF and LVF Do not Show a Contralateral Bias in the Deaf PAC and Association Cortex
Interestingly, despite a right-hemisphere advantage for early deaf participants in the STS region, and hinted at in the PAC, there was more activation overall to stimuli presented in the RVF. Behaviorally, an RVF advantage for visual motion perception has often been reported for early deaf participants (e.g., direction of motion: Neville & Lawson, 1987; direction of motion: Bosworth & Dobkins, 1999; motion velocity: Brozinsky & Bavelier, 2004; direction of motion: Bosworth et al., 2013; see also Samar & Parasnis, 2007; but see Hauthal et al., 2013, for an LVF advantage for movement localization in late signers).
A right-hemisphere advantage has been reported in previous neuroimaging studies investigating visual motion or flickering stimulus responses in early deaf people: In the auditory cortices, the right hemisphere was dominantly (Finney et al., 2003) or exclusively (Fine et al., 2005; Finney et al., 2001) activated. The right-hemisphere advantage reported here is also in line with the finding that only the right auditory cortex (planum temporale) showed a correlation between increased cortical thickness and enhanced visual motion detection thresholds in the study on early deaf people of Shiell et al. (2016). The right auditory cortex was also shown with ERP source localization to be dominant in hearing-restored deaf individuals when viewing visual stimuli (Sandmann et al., 2012). Although some studies have reported left-hemisphere advantages in early deaf people, such effects either could not be localized (e.g., Neville & Lawson, 1987) or were relatively small, sometimes nonsignificant effects, reported in hMT+ (e.g., Fine et al., 2005; Bavelier et al., 2001), which in our study, on direction-selective responses, did not show significant differences across participant groups but an appearance of right lateralization for early deaf participants only (see Figures 2B and 4B).
Here, we can reconcile behavioral RVF and neural right-hemisphere advantages in response to directional motion: The remapped auditory cortices do not show a strong contralateral bias like other direction-selective areas, for example, hMT+; instead, the right hemisphere dominates regardless. As addressed in the first section of the Discussion, the STS region possesses large spatial fields and, particularly in the right hemisphere, low sensitivity to retinotopic organization (e.g., Saygin & Sereno, 2008; see also Almeida et al., 2015; Grossman & Blake, 2001). The recruited association cortex seems to be the best candidate for behavioral RVF advantages, because this region showed the most extensive changes between early deaf and hearing participants here. In addition, in a previous study reporting effects in hMT+, the posterior STS was shown to be 9.3 times larger in size (in comparison, hMT+ was only 1.08 times larger) and seven times greater in percent signal change (hMT+: 1.05 times greater) in deaf than hearing participants in response to attended velocity of visual motion (see Table 4 of Bavelier et al., 2001).
A right-hemisphere advantage paired with an RVF advantage goes against the assumption that neural activation to visual stimuli is necessarily contralateral, which has frequently been made in the literature on neural plasticity in early deaf people (e.g., Bavelier et al., 2001; tentatively in Bosworth et al., 2013; Hauthal et al., 2013; Brozinsky & Bavelier, 2004; Bosworth & Dobkins, 1999, 2002). In a study reporting a left-hemisphere advantage with attention-related modulation of ERPs to peripheral visual targets in early deaf people, Neville and Lawson (1987) hypothesized that the left hemisphere was remapped for sign language processing and therefore could have different sensitivities to stimuli such as visual motion or stimulus localization. However, it is not clear that the left hemisphere is specialized for sign language processing in early deaf people: Deaf and early-signing hearing participants have been shown to have bilateral (STS) activation to sign language; and early deaf participants, to have more right STS activation to written language (e.g., Sadato et al., 2004; Neville et al., 1998). Our finding of a right-hemisphere advantage compatible with an RVF advantage offers an alternative explanation and unites most neural and behavioral findings regarding motion perception of early deaf people.
Speculatively, our findings suggest that behavioral advantages in early deaf people, particularly for motion discrimination in the RVF, may be supported by increased STS region activation in the right hemisphere (again, see Figure 4A). Although limited by a small sample size, five of our six deaf participants also participated in a behavioral study in our laboratory, in which thresholds on the percent dot coherence required for direction of visual motion in the LVF and RVF were acquired. We found a suggestive correlation with this measure of behavioral direction discrimination ability (i.e., lower percent dot motion coherences required) and the extent of activation in the right STS region, R2 = .30, although not significant, p = .10. In comparison, the extent of bilateral hMT+ activation showed no correlation, R2 = .01, p = .78. However, this tentative result would need to be confirmed with larger sample sizes in future studies. At the least, we are able to introduce the hypothesis that the auditory and association cortices in early deaf individuals are sensitive to directional visual motion and that this neural reorganization may support a behavioral advantage reported previously for visual motion direction discrimination.
Limitations and Future Directions
One limitation of this study was that the sample consisted of only six participants per group: Although the results were reliable across individual participants, they would be strengthened by replication in future studies, potentially with larger sample sizes. Another limitation here is that an eye tracker was not used during the fMRI experiment to ensure fixation. However, it is unlikely that eye movements could explain the results. The functionally defined regions were localized with centrally presented stimuli, and the differences between deaf and hearing participants for peripherally presented stimuli were highly specific (e.g., the enhanced STS region recruitment was restricted to the right hemisphere and RVF). In addition, cues for participants to report the direction of motion were given at random intervals throughout the experiment, such that they were neither periodic nor associated with directional motion presentation times (see Methods). A third limitation of this study was that the STS region was defined broadly in each participant; future studies could use anatomical landmarks or more specific functional localizers in hearing participants to demarcate more precise subregions. Finally, in this study, directional visual motion coincided with coherent visual motion. Although it may be argued that coherent motion inherently possesses directionality, future studies may address the influence of coherency on directional motion responses (see Braddick et al., 2008).
This research was supported by grants from the National Institutes of Health (NIH; grants EY023268 to F. J. and P20 GM103650 to M. A. W.). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. Talia L. Retter is supported by the Belgian National Foundation for Scientific Research (grant FC7159). The authors are thankful to Andrea Conte and Bruno Rossion for access to the stimulation program XPMan, Revision 111, as well as to Xiaoqing Gao for his help with the frequency domain analysis, and O. Scott Gwinn for use of his behavioral data on discrimination thresholds for the deaf participants as well as help with stimulus generation.
Reprint requests should be sent to Talia L. Retter, Psychology and Integrative Neuroscience, University of Nevada, Reno, 1664 N. Virginia St., Reno, NV 89503, or via e-mail: firstname.lastname@example.org.
This paper is part of a Special Focus deriving from a symposium at the 2017 International Multisensory Research Forum (IMRF).