Abstract

The integration of information from multiple senses leads to a plethora of behavioral benefits, most predominantly to faster and better detection, localization, and identification of events in the environment. Although previous studies of multisensory integration (MSI) in humans have provided insights into the neural underpinnings of MSI, studies of MSI at a behavioral level in individuals with brain damage are scarce. Here, a well-known psychophysical paradigm (the redundant target paradigm) was employed to quantify MSI in a group of stroke patients. The relation between MSI and lesion location was analyzed using lesion subtraction analysis. Twenty-one patients with ischemic infarctions and 14 healthy control participants responded to auditory, visual, and audiovisual targets in the left and right visual hemifield. Responses to audiovisual targets were faster than to unisensory targets. This could be due to MSI or statistical facilitation. Comparing the audiovisual RTs to the winner of a race between unisensory signals allowed us to determine whether participants could integrate auditory and visual information. The results indicated that (1) 33% of the patients showed an impairment in MSI; (2) patients with MSI impairment had left hemisphere and brainstem/cerebellar lesions; and (3) the left caudate, left pallidum, left putamen, left thalamus, left insula, left postcentral and precentral gyrus, left central opercular cortex, left amygdala, and left OFC were more often damaged in patients with MSI impairments. These results are the first to demonstrate the impact of brain damage on MSI in stroke patients using a well-established psychophysical paradigm.

INTRODUCTION

Neurophysiological studies in animal models have demonstrated the existence of multisensory neurons, which respond to stimulation of more than one sense (e.g., Stein & Stanford, 2008; Stein & Meredith, 1993). For one type of these multisensory neurons, the response to multisensory stimulation is larger than the sum of the unisensory responses, indicating that these “superadditive neurons” integrate information from different senses.1 In recent years, an overwhelming amount of behavioral, neurophysiological, and neuroimaging evidence for the presence of multisensory integration (MSI) in humans has surged (Van der Stoep, Nijboer, Van der Stigchel, & Spence, 2015; Leone & McCourt, 2013; Fiebelkorn, Foxe, Butler, & Molholm, 2011; Alais & Burr, 2004; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Talsma, Doty, & Woldorff, 2007; Molholm et al., 2002, 2006; Gondan, Niederhaus, Rösler, & Röder, 2005; Lovelace, Stein, & Wallace, 2003; Ernst & Banks, 2002; Calvert, Campbell, & Brammer, 2000; Hughes, Reuter-Lorenz, Nozawa, & Fendrich, 1994). For example, because of MSI, multisensory stimuli in the environment can be detected, localized, and identified faster and more accurately than the unisensory component stimuli (Calvert, Spence, & Stein, 2004; Spence & Driver, 2004). As such, MSI is a crucial function for optimal perception of and interaction with the environment.

There is no brain region that is solely responsible for MSI (Ghazanfar & Schroeder, 2006; Calvert & Thesen, 2004). Instead, a broad network of cortical and subcortical brain networks has been associated with integrating information from different senses. Among others, areas such as the superior colliculus, thalamus, amygdala, claustrum, insula, STS, and the lateral intraparietal area have been shown to be involved in integrating auditory, visual, and/or tactile input (Stevenson & James, 2009; Ghazanfar & Schroeder, 2006; Molholm et al., 2006; Calvert & Thesen, 2004; Wallace, Meredith, & Stein, 1998; Meredith & Stein, 1986, 1996; Meredith, Nemitz, & Stein, 1987). Furthermore, the frontal–parietal network is involved in visuotactile and audiotactile integration in peripersonal space (the regions of space immediately surrounding our body), which allows us to respond faster to information close to our body (parts; Van der Stoep, Serino, Farnè, Di Luca, & Spence, 2016; Van der Stoep, Nijboer, et al., 2015; Serino et al., 2015; Serino, Canzoneri, & Avenanti, 2011; Làdavas & Farnè, 2004; Farnè & Làdavas, 2002; Graziano & Gross, 1998; Fogassi et al., 1996; Gross & Graziano, 1995; Graziano, 1994). Besides these multisensory brain areas, direct connections between unisensory brain regions also contribute to interactions between the senses (Ghazanfar & Schroeder, 2006). From these observations, it is clear that MSI is a widely distributed process that can be observed in many brain areas (Ghazanfar, & Schroeder, 2006; Calvert & Thesen, 2004).

So far, neuroimaging and EEG studies have revealed the neural substrates that are activated by multisensory stimuli and the temporal development of this activation (e.g., Tyll et al., 2013; Nath & Beauchamp, 2011; Senkowski, Saint-Amour, Höfle, & Foxe, 2011; Cappe, Thut, Romei, & Murray, 2010; Stevenson & James, 2009; Martuzzi et al., 2007; Molholm et al., 2002; though see Stevenson et al., 2014; Murray, Cappe, Romei, Martuzzi, & Thut, 2012; Calvert & Thesen, 2004, for a discussion of whether MSI effects observed in BOLD signals and ERPs reflect true integration). These studies cannot, however, demonstrate a causal link between specific brain regions and MSI in humans. TMS has provided researchers with the opportunity to investigate such a causal link by temporarily deactivating certain brain areas. This way, it has been shown, for example, that the STS plays a crucial role in audiovisual integration (Beauchamp, Nath, & Pasalar, 2010). Unfortunately, reports on the link between damaged brain regions and behavioral indices of MSI are scarce. Investigating the damaged brain may provide novel information on the relationship between certain brain areas and behavioral indices of MSI.

Although several cases have been reported describing impairments in or alterations of MSI after brain damage, such reports are rare and do not allow the identification of crucial brain areas using lesion overlay analysis given that these are studies of single patients (see Cecere, Romei, Bertini, & Làdavas, 2014; Freeman et al., 2013; Baum, Martin, Hamilton, & Beauchamp, 2012; Hamilton, Shenton, & Coslett, 2006, for case studies of audiovisual integration after brain damage; see also Oliveira et al., 2011, for a study of visual, vestibular, and somatosensory integration in balance control after stroke). Furthermore, studies involving groups of patients with brain damage have mainly focused on whether patients could benefit from multisensory stimulation (Passamonti, Bertini, & Làdavas, 2009; Làdavas, 2008; Frassinetti, Bolognini, Bottari, Bonora, & Làdavas, 2005; Làdavas & Farnè, 2004; Farnè & Làdavas, 2002; see Tinga et al., 2016, for a review). However, observing an improvement in performance during multisensory stimulation does not necessarily mean that these patients could actually integrate information from multiple senses as stimulating multiple senses can also enhance behavior via statistical facilitation. That is, performance on a task can improve during multisensory stimulation even if the signals are independently processed and not integrated (Miller, 1982, 1986, 2016; Ulrich, Miller, & Schröter, 2007).

In the current study, the association between MSI after stroke and location of the lesion was investigated by adopting a combination of behavioral measures of MSI and lesion analysis in a group of patients who suffered an ischemic stroke. MSI was quantified using the redundant target task (Van der Stoep, Van der Stigchel, Nijboer, & Spence, 2017; Van der Stoep, Van der Stigchel, Nijboer, & Van der Smagt, 2016; Nardini, Bales, & Mareschal, 2015; Van der Stoep, Spence, Nijboer, & Van der Stigchel, 2015; Van der Stoep, Van der Stigchel, & Nijboer, 2015; Girard, Pelland, Lepore, & Collignon, 2013; Girard, Collignon, & Lepore, 2011; Ulrich et al., 2007; Hughes et al., 1994; Miller, 1982, 1986). In this task, participants have to respond to the onset of a sound, a light, or their combination. Typically, RTs in the audiovisual condition (combined targets) are faster than in the fastest unisensory condition (auditory or visual only). Comparing RTs in the multisensory condition to RTs in the unisensory conditions provides information on the amount of multisensory response enhancement (MRE). Although this multisensory benefit may, at first sight, be interpreted as the result of MSI, a speed-up of responses in the multisensory condition can also be observed without MSI as the underlying cause. Each unisensory stimulus (auditory, visual) is detected with a certain speed, and there is some variability in this speed of detecting the stimulus. During audiovisual trials, these two separate signals (auditory, visual) are presented together and race to reach a certain response threshold. The first signal to reach this threshold triggers a response. Consequently, the probability of a fast RT increases in the audiovisual condition, simply because the probability of a fast response in the unisensory conditions can be summed. This is called “statistical facilitation” (see Miller, 2016; Raab, 1962). The underlying cause of the speed-up in the multisensory condition can be investigated by testing the race model inequality (RMI; Miller, 1982). With this test, audiovisual RTs are compared with how fast responses can become based on statistical facilitation using RTs from the unisensory conditions. If RTs in the audiovisual condition are faster than what can be expected based on statistical facilitation (i.e., RMI violation), then it can be assumed that auditory and visual information is integrated by the brain. In this study, impairments in MSI are defined as lack of RMI violation at a group level and as a significant decrease in RMI violation relative to the control group at an individual level. MSI impairments were related to damage in certain brain areas using lesion subtraction analysis.

Based on previous studies (Yau, DeAngelis, & Angelaki, 2015; Ghazanfar & Schroeder, 2006; Calvert & Thesen, 2004), we hypothesized that lesions to various subcortical and cortical areas could result in impairments in MSI. For example, areas like the thalamus, amygdala, insula, STS, superior parietal lobule, and/or the superior colliculus could lead to MSI deficits. Regarding hemispheric laterality, both the left and right STS and SPL are involved in MSI. However, some multisensory brain regions in the left hemisphere (left STS, left insula) may be more strongly involved in audiovisual integration than the right hemisphere (e.g., Calvert et al., 2000). We presented auditory (A), visual (V), and audiovisual (AV) targets in both the left and right visual hemifield. This way, impairments in MSI could be measured behaviorally for the contralesional and ipsilesional visual hemifield and analyzed in relation to lesion location using a qualitative lesion analysis. Analyzing overlapping lesion locations for patients with MSI impairment allowed investing which brain regions are responsible for MSI in the contralateral, ipsilateral, or both visual hemifields. This study also allowed us to determine the severity and specificity of MSI impairments in the chronic phase after stroke.

METHODS

Participants

The patient group consisted of 21 patients (17 men, Mage = 57.81 years, SD = 9.27) who were selected from a database that was available at the University Medical Center Utrecht containing information on patients who were admitted to the stroke unit and received treatment for stroke between 2009 and 2014. All patients were in the chronic phase after stroke and invited to take part in the current study, which took place in the lab of Experimental Psychology at Utrecht University. Inclusion criteria were as follows: (1) ischemic stroke as diagnosed by a neurologist, (2) signed informed consent, (3) no severe deficits in communication or verbal comprehension, (4) normal or corrected-to-normal visual acuity and hearing, (5) aged between 18 and 85 years, and (6) able to perform the tasks (i.e., use the computer mouse, a detection rate of ≥0.80, and being able to respond to targets in the RT task within the response window of 2.1 sec). Exclusion criteria for the analyses were (1) visuospatial neglect (deviating performance on a digital line bisection task indicative of visuospatial neglect; cutoff deviation < −0.81° or > 0.51° from the center2), (2) hemianopia, and (3) no brain damage visible on the brain scan.

Additionally, 14 healthy participants (seven men, Mage = 56.50 years, SD = 18.8) acted as a control group to the group of participants with brain lesions. A statistically significant reduction in RMI violation relative to the control group using single-case statistics was used to determine MSI impairments (see below). All participants in the control group reported normal or corrected-to-normal visual acuity and normal hearing, except for three participants who reported having a hearing loss. Two participants used hearing aids, and one participant was seeing a physician for potential treatment. Hearing loss can potentially affect sound detection, sound localization, and MSI. However, all volunteers who reported hearing loss performed the detection task with a minimum of 98% correct and had RTs that were within 1.5 times the interquartile range of the control group in all conditions (A, V, AV, and catch trials). Furthermore, excluding these three control participants from the control group resulted in the same number of patients with MSI impairments in the case–control analysis (n = 7, 33% of the patient sample). Therefore, all 14 participants were included in the analysis.

All participants gave written informed consent according to the standards of the Declaration of Helsinki. The study protocol, information letter, and informed consent forms were approved by the medical ethical review board of the University Medical Centre Utrecht (Research Protocol No. 10-415/E).

Apparatus

Visual stimuli were projected on a white projection screen (50 × 75 cm) using an Acer X1261P projector with a refresh rate of 60 Hz. The projector was placed above and behind the participant projecting down on the projection screen, which was placed at ∼87 cm from the participant. The projection screen was sound transparent, allowing sound waves to pass through the projection screen. Auditory (A) stimuli were presented using two speakers that were placed behind the projection screen aligned with the projection of the visual stimulus locations (Harman/Kardon HK206, frequency response: 90–20,000 Hz). A custom-made response box was used to collect responses. The experiment was programmed in MATLAB (The MathWorks, Inc.) using Psychtoolbox (Brainard, 1997).

Stimuli, Procedure, and Design

All participants were seated in a dimly lit lab with the setup placed in the center of the room. Each trial, a black fixation cross (1° × 1°, 4.87 cd/m2) was presented on a gray background (102 cd/m2) at eye height in the center of the screen for 750–1250 msec. After the disappearance of the fixation cross, a blank screen was presented for a random duration of between 200 and 250 msec after which an A, V, or AV target was presented for 100 msec at 16° to the left or to the right of the central fixation cross or no target was presented (catch trial). Visual (V) stimuli were white-filled circles (3.3° × 3.3°, 236 cd/m2). Auditory (A) stimuli consisted of 100-msec white noise bursts (with a 15-msec linear rise and fall of the signal) that were presented at ∼80 dB(A). The audiovisual (AV) stimuli were a combination of the A and V stimulus and were always presented spatially and temporally aligned to the left or right side of the fixation cross. Temporal alignment of sound and light onset was confirmed using an oscilloscope. The response window from target onset (which was also the intertrial interval) was set to 2100 msec.

In total, there were 240 target present trials (evenly distributed across left and right locations and A, V, AV targets) and 30 target absent trials (30 trials, 11%). Target type (A, V, AV, Catch) and target location were randomized. Participants had to respond to unisensory (A, V) or spatially and temporally aligned AV targets that appeared to the left or right of a central fixation cross using a single button on a custom-made response box that was connected to a computer (Van der Stoep, Van der Stigchel, et al., 2015). Participants could respond with their preferred hand. When no target was presented, participants had to withhold their response. There were seven practice trials to let participants become familiar with the task.

Outcome Measures

Age, sex, time poststroke, hand preference, lesion side, scan type, and lesion classification were obtained from the database.

The proportion of hits was calculated for all target present conditions and the proportion of correctly withheld responses for catch trials. A hit was defined as a button press after target onset and before the end of each trial. The proportion of hits and withheld responses was based on all trials (before filtering RTs, see below).

RTs shorter than 100 msec or longer than 2100 msec (end of the trial) were removed from further RT analysis as they were assumed to be the result of either anticipation or not paying attention to the task. The median rather than the mean RT of all trials in each condition was used as a measure of central tendency per participant, given that the RT distributions in our sample were skewed. Given that each condition contained the same number of observations and the proportion of hits was very high in all conditions, the median instead of the mean RT was used as a measure of central tendency with the advantage that the median is less affected by deviations from normality than the mean (see Whelan, 2008, for more information on analyzing RTs).

Participants typically respond faster to audiovisual target than to unisensory targets (auditory or visual alone). As explained in the Introduction, this MRE can also occur in the absence of MSI. To investigate whether MRE was due to MSI or statistical facilitation (independent processing of sensory input), RT distributions were analyzed using a RMI analysis. For each participant and each condition, the cumulative distribution functions (CDFs) were used. Each CDF describes the probability of a certain RT (y-axis) for a range of time points (x-axis) and is based on the observed RTs. This probability ranges from 0 to 1, with 1 indicating that 100% of the RTs in a condition is faster than a certain time. The AV CDF was compared with the sum of the unisensory CDFs (A only + V only). This allowed us to test the RMI (see Gondan & Minakata, 2016; Miller, 1982, 1986, 2016; Ulrich et al., 2007; Raab, 1962):
PRTAV<tPRTA<t+PRTV<t
(1)
The inequality describes the probability (P) that a given RT in the AV condition (RTAV) is shorter than or equal to the sum of the probability of that RT in the unisensory conditions (RTA, RTV) for every time point (t), with t > 0. To test for violations of the inequality, the race model CDF was subtracted from the audiovisual CDF. This created a difference curve, with positive values indicating that AV RTs were faster than statistical facilitation and negative values indicating that the audiovisual RTs could be explained by statistical facilitation (see Figure 2 for RMI violations). At a group level, violations of the inequality were statistically analyzed using Bonferroni-corrected one-sample t tests at multiple points along the differences curve (the 10th to the 90th percentile, in steps of 10%). Mann–Whitney U tests were used instead of one-sample t tests when the assumption of normality was violated. One-sample t tests were preferred when the data were normally distributed because they generally have more statistical power. Significant positive differences from zero at the group level at any point along the RT distribution are indicative of MSI. Lack of RMI violation at larger percentiles (e.g., 70th to 90th) is very common and can be explained by the deviating shape of the summed CDF (CDFA + CDFV; e.g., see Van der Stoep, Van der Stigchel, et al., 2016; Van der Stoep, Van der Stigchel, et al., 2015; Girard et al., 2011; Miller, 1982). Summing the auditory and visual CDF creates a curve that ranges from 0 to 2. Probabilities larger than 1 do not make sense so the summed CDF is cut off at 1 (also see Gondan & Minakata, 2016, Figure 1; Miller, 1982, Figure 2).
To quantify the amount of MRE for the different patient groups, the relative MRE (rMRE) was calculated using the formula:
rMRE=minmedianRTAmedianRTVmedianRTAVminmedianRTAmedianRTV×100%
(2)
The rMRE indicates how much faster responses in the AV condition are relative to the fastest median response in the unisensory conditions, expressed as a percentage. Positive values indicate multisensory enhancement; negative values indicate multisensory interference. The relative instead of the absolute amount of MRE was chosen because participants (healthy controls and patients) could differ in their absolute RTs (i.e., patients often have somewhat slower responses). Using a relative measure allowed for a more standardized way of quantifying the multisensory benefits for the control and patient groups.

Generation of Lesion Maps

All lesion data were anonymized before analysis. Manual segmentation of infarcts on transversal slices of CT (n = 12), FLAIR (n = 8), or T1 MRI sequences (n = 1) was performed using MRIcron (version May 2016, see Rorden, Karnath, & Bonilha, 2007) by a trained rater (J. B. M.) who was blinded to the behavioral data. Segmentations were transformed to the MNI-152 template (Klein, Staring, Murphy, Viergever, & Pluim, 2010; Fonov, Evans, McKinstry, Almli, & Collins, 2009). This lesion map generation procedure was the same as in previous studies (for more details, see Biesbroek, Weaver, & Biessels, 2017; Ten Brink, Verwer, Biesbroek, Visser-Meily, & Nijboer, 2017; Zhao et al., 2017). A rigorous quality check of the registration results was performed by comparing the original scan and the lesion map in MNI space, followed by a manual adjustment to correct for slight registration errors if necessary. Total normalized infarct volume was calculated using the lesion maps in standard (MNI) space. The use of normalized lesion maps in standard space serves as a correction for differences in intracranial volume. Voxel volume (in all cases 1 mm3) was multiplied with binary lesion status (i.e., 1 when the voxel is lesioned, 0 when the voxel is not lesioned) and divided by 1000, so that volume was expressed in milliliters.

Analyses

The control group and the patient group were compared in terms of age and handedness to check whether the two groups were comparable. Task performance (hits, false alarms) was analyzed for each patient group (left hemisphere lesion, right hemisphere lesion, no cortical lesion) and the control group. The RT data were analyzed at two levels.

First, group analyses were performed to compare the control group and the left and right hemisphere patient group in terms of RTs and RMI violation. The amount of positive RMI violation at each quantile (10–90% in steps of 10%) was tested against zero using one-sample t tests or one-sample Wilcoxon signed-rank tests (depending on whether or not the data were normally distributed). This was done to investigate whether MSI occurred within each group, to investigate how robust MSI is to cortical lesions in general, and whether the side of the lesion mattered (left vs. right hemisphere). The group without cortical lesions was excluded for this analysis because the group was very small (n = 3), and we had no clear hypotheses regarding lateralization for brainstem and cerebellar lesions. The amount of RMI violation was compared between groups (control, left hemisphere, right hemisphere group) using a mixed ANOVA.

Given the large number of brain areas involved in MSI, patients could have more or fewer difficulties with integrating auditory and visual input depending on the lesion location. Therefore, the group analysis only paints a general picture of what is happening with MSI in the patient groups. To investigate MSI impairments in more detail, the patient group was divided into an MSI impaired and unimpaired group by comparing RMI violations of each patient to the control group using single-case statistics. For each patient and each of the nine quantiles in a condition, it was tested whether the RMI violation differed from the control group using Crawford and Howell's modified t test (alpha = .05, Crawford & Garthwaite, 2005). For each patient–control comparison, the p values were Bonferroni-corrected for nine comparisons given that we compared each case with the control group at nine quantiles in each visual hemifield. Negative deviations of RMI violation (i.e., patients' RMI violation < control group RMI violation) indicate reduced MSI relative to the control group in the left and/or right visual hemifield. Patients who had significantly less RMI violation than the healthy control group for at least one quantile in a region of space (left, right, or both hemifields) were labeled as MSI impaired. Based on this indication, patients were assigned to an MSI impaired and not impaired group. This single-case analysis categorized a patient as MSI impaired if one of the nine quantiles deviated significantly from the patient group in the left or right visual hemifield. To check the validity of this MSI impairment categorization method, we also compared the average RMI violation across all quantiles between each participant and the control group for each visual hemifield using Crawford and Howell's modified t tests. This led to the same categorization for all but one patient with cerebellar lesions, who was identified as having unimpaired MSI using the second approach. Choosing the first or second categorization method did not affect the critical areas identified by the lesion subtraction analysis. We therefore used the first method to categorize patients but also provide the categorization outcome based on the second method in Table 2. Furthermore, the impact of an impairment of MSI on behavior was quantified by calculating the rMRE for the two patient groups. For each group, it was tested whether rMRE was still significantly different from zero using one-sample t tests.

A lesion overlay plot including all 21 patients was generated to provide an overview of the lesion locations in the patient group (see Figure 1A). Lesions overlay plots were also generated for the three main groups of lesions: cortical lesions in the left hemisphere (Figure 1B), right hemisphere (Figure 1C), and subcortical lesions (Figure 1D). Patients in the left hemisphere group had no brain damage in the right hemisphere and vice versa. This analysis was done to provide an overview of the overlap in cortical lesions in each hemisphere group and in the subcortical lesion group.

Figure 1. 

Lesion overlay plots for (A) all patients, (B) the left hemisphere group, (C) the right hemisphere group, and (D) the group without cortical lesions. Voxels that are damaged in at least one patient are projected on the 1-mm MNI-152 template. The colored bars indicate the number of patients with a lesion in each voxel.

Figure 1. 

Lesion overlay plots for (A) all patients, (B) the left hemisphere group, (C) the right hemisphere group, and (D) the group without cortical lesions. Voxels that are damaged in at least one patient are projected on the 1-mm MNI-152 template. The colored bars indicate the number of patients with a lesion in each voxel.

To identify the critical lesion locations related to impairments in MSI, a lesion subtraction analysis was performed by generating (1) a lesion overlay of patients who showed no MSI impairment (Figure 3A), (2) a lesion overlay of patients with MSI impairments (Figure 3B), and (3) a lesion subtraction of patients with and without MSI impairment to identify critical lesion locations (Figure 3C). We used the MSI impairment categorization based on the per quantile RMI violation case–control comparison to define the impaired and unimpaired group. The lesion subtraction analysis identifies brain regions that are more often damaged in patients with an impairment in MSI and spared in patients without MSI impairment. Although statistical lesion analysis (i.e., voxel-wise statistical testing) was not possible due to the limited number of patients, we used a (somewhat arbitrary) cutoff of 20% to highlight the brain regions that are most clearly related to MSI impairment. The 20% cutoff refers to the minimum difference in the percentage of patients with lesions in a certain voxel between the MSI impaired and unimpaired group. Brain regions in which voxels in the lesion subtraction plot exceeded the 20% cutoff criterion were identified using the Oxford–Harvard cortical and subcortical atlas (Desikan et al., 2006). This lesion subtraction analysis identifies brain regions that are damaged in patients with MSI impairment and spared in patients with no MSI impairment.

The Greenhouse–Geisser correction was used to adjust the degrees of freedom whenever the assumption of sphericity was violated. Nonparametric tests were used whenever data were not normally distributed (e.g., indicated by the reported U and Z values of the Mann–Whitney U and one-sample Wilcoxon signed-rank test, respectively). The p values were Bonferroni-corrected when required.

RESULTS

Demographic and Stroke Characteristics

The control group and the patient group did not differ in terms of age, t(17.256) = −.242, p = .812, sex, χ2(1) = 1.652, p = .199, or handedness (U = 144, p = .802). All patients had unilateral lesions (11 left hemisphere lesions, 7 right hemisphere lesions), and three patients had brainstem and/or cerebellar lesions (labeled “no cortical lesions”). See Table 1 for an overview of the characteristics of the two groups.

Table 1. 
Characteristics of the Patient and Control Groups (SD or % between Brackets)
 Control GroupPatient Group
Group size 14 21 
Sex (%) 
 Male 7 (50%) 17 (65%) 
Average age in years 56.50 (18.81) 57.81 (9.27) 
Average handedness scorea 9.86 (0.54) 9.05 (4.36) 
Average time poststroke onset (weeks) – 167.07 (82.15) 
Stroke type (%)b 
 TACI  1 (4.8%) 
 PACI  12 (57.1%) 
 LACI  3 (14.3%) 
 POCI  5 (23.8%) 
Lesion location 
 Left hemisphere – 11 (52.4%) 
 Right hemisphere – 7 (33.3%) 
 No cortical lesion – 3 (14.3%) 
Average total lesion size (mm3– 54.75 (15.55) 
 Control GroupPatient Group
Group size 14 21 
Sex (%) 
 Male 7 (50%) 17 (65%) 
Average age in years 56.50 (18.81) 57.81 (9.27) 
Average handedness scorea 9.86 (0.54) 9.05 (4.36) 
Average time poststroke onset (weeks) – 167.07 (82.15) 
Stroke type (%)b 
 TACI  1 (4.8%) 
 PACI  12 (57.1%) 
 LACI  3 (14.3%) 
 POCI  5 (23.8%) 
Lesion location 
 Left hemisphere – 11 (52.4%) 
 Right hemisphere – 7 (33.3%) 
 No cortical lesion – 3 (14.3%) 
Average total lesion size (mm3– 54.75 (15.55) 

TACI = total anterior circulation infarct; PACI = partial anterior circulation infarct; LACI = lacunar circulation infarct; POCI = posterior circulation infarct.

a

Handedness was quantified using the Dutch handedness questionnaire (van Strien, 2003). A score of −10 indicates extreme left handedness, and a score of 10 indicates extreme right handedness.

b

Classification using the Oxfordshire Community Stroke Project Classification (Bamford, Sandercock, Dennis, Burn, & Warlow, 1991).

Redundant Target Task

Task Performance

Patients in the left hemisphere group (n = 11) performed very well in terms of target detection and response inhibition. The average proportion of detected targets in the left (MA = 0.991, SE = 0.005; MV = 0.984, SE = 0.005; MAV = 0.986, SD = 0.008, range = 0.93–1) and right visual hemifield was very high (MA = 0.991, SE = 0.004; MV = 0.977, SE = 0.009; MAV = 0.977, SE = 0.011, range = 0.90–1). On average, this patient group rarely responded on catch trials (MCatch = 0.036, SD = 0.020, range = 0.00–0.20).

Patients in the right hemisphere group (n = 7) also performed very well, both for targets in the left (MA = 0.996, SE = 0.004; MV = 1, SE = 0; MAV = 0.996, SD = 0.004, range = 0.98–1.00) and the right visual hemifield (MA = 1, SE = 0; MV = 1, SE = 0; MAV = 0.996, SD = 0.004, range = 0.98–1). None of the patients in this group responded to catch trials.

Task performance of patients with subcortical lesions (no cortical lesions) was also good (n = 3, left targets: MA = 1, SE = 0; MV = 0.958, SE = 0.0417; MAV = 1, SE = 0, range = 0.88–1; right targets: MA = 0.983, SE = 0.167; MV = 0.933, SE = 0.067; MAV = 0.992, SE = 0.008, range = 0.80–1; MCatch = 0.011, SE = 0.011, range = 0.00–0.03).

Overall, the performance of the patient group was very similar to that of the control group. The participants in the control group (n = 14) responded to a very high proportion of targets in the target present conditions (left targets: MA = 0.998, SE = 0.002; MV = 0.977, SE = 0.012; MAV = 0.996, SE = 0.002, range = 0.85–1.00; right targets: MA = 0.996, SE = 0.002; MV = 0.982, SE = 0.008; MAV = 0.998, SE = 0.002, range = 0.90–1) and rarely responded on catch trials (MCatch = 0.010, SE = 0.004, range = 0.00–0.03).

Given the high proportion of detection and the low number of responses on catch trials in all conditions in both groups, we did not further analyze the proportion of hits and false alarms.

RTs

A 2 × 3 × 2 mixed repeated-measures ANOVA with the within-subject factors Visual hemifield (left, right) and Target modality (A, V, AV) and the between-subject factor Group (control, left hemisphere, right hemisphere lesion) was used to analyze RTs. The group without cortical lesions (only subcortical brain damage) was not included in the analysis because of the low number of patients in this group and the lack of lesion lateralization in these patients.

There was a main effect of Target modality, F(2, 29) = 35.632, p < .001, ηp2 = .544. Responses to AV targets (M = 314 msec, SE = 23) were faster than to A (M = 375, SE = 25) and V (M = 381, SE = 20) targets (all ps < .001, Bonferroni-corrected). There was no significant difference in RTs between the A and V conditions (p > .05). Additionally, there was a main effect of Group, F(2, 29) = 3.574, p = .041, ηp2 = .198. On average, the responses of patients with a lesion in the left hemisphere (M = 430 msec, SE = 50) were slower than responses of the control group (M = 302 msec, SE = 19, p = .04), but their RTs did not differ from the right hemisphere lesion group (M = 337 msec, SE = 28). The right hemisphere lesion group did not differ from the control group (p = 1). Furthermore, there was no main effect of visual hemifield, F(1, 33) = 0.446, p = .509, nor were there any two- or three-way interactions between target modality, visual hemifield, and group (0 < all Fs < 3.4, all ps > .05).

Although the left hemisphere group was somewhat slower, overall, participants had faster responses in the AV condition relative to the unisensory conditions, and RTs in the unisensory conditions (A, V) did not differ. See Figure 1A, B, and C for the average of median RTs in each condition and group.

RMI Violation

Multisensory stimulation can lead to faster responses even in the absence of MSI (i.e., independent processing) due to statistical facilitation. To test whether the observed speed-up of responses in the audiovisual condition could be explained by statistical facilitation or MSI, the RMI was tested in the control group and the left and right hemisphere patient groups for targets in the left and right visual hemifield separately (Ulrich et al., 2007; Miller, 1982, 1986). This allowed us to test whether MSI occurred in each group and whether there were differences between the left and right visual hemifield. Importantly, if RMI violation occurred at any of the percentiles, MRE cannot be explained by independent processing of sensory input.

In the control group, we observed RMI violations from the 10th to the 60th percentile in the left andright visual hemifield (all ts > 4, p < .05; see Figure 1D and E). This indicates that the observed speed-up of responses in the audiovisual condition could not be explained by statistical facilitation, which is indicative of MSI (see Figure 1D and E).

In the left hemisphere patient group, there were no significant positive RMI violations in the left or right visual hemifield (all Zs < 2, p > .05). In the right hemisphere patient group, significant positive RMI violations were observed at the 40th percentile in the left visual hemifield, t(6) = 4.280, p = .05, and the 30th and 40th percentile in the right visual hemifield (all ts > 4, all ps < .05).

The results from these one-sample t tests indicate that the control group and the right hemisphere lesion group integrated auditory and visual information in the left and right visual hemifield. In contrast, the left hemisphere group did not integrate auditory and visual information in the left or right visual hemifield. The amount of quantiles at which significant RMI violations were observed differed between the right hemisphere and control groups, suggesting that MSI may be reduced in the right hemisphere group relative to the control group.

The amount of RMI violation was compared between groups using a mixed ANOVA with within-subject factors Visual hemifield (left, right) and Quantile (10th to 90th) and the between-subject factor Group (control, left hemisphere, right hemisphere group). There was a main effect of Quantile, F(1.433, 41.559) = 22.398, p < .001, ηp2 = .436, ε = .179, indicating that RMI violations decrease at later quantiles (see Methods for an explanation of this common phenomenon). The interaction between Group and Quantile was significant, F(16, 121) = 3.017, p = .043, ηp2 = .172, reflecting decreased RMI violation with increasing percentiles in the left hemisphere group relative to the right hemisphere and control groups (see Figure 2). However, there was no main effect of Group, no main effect of Visual hemifield, nor any interaction between Visual hemifield and Group or Quantile (all Fs < 1, all ps > .05).

Figure 2. 

The average of the median RTs in the A, V, and AV conditions in the left and right visual hemifield for each group (A, B, C). RMI violations in the left (D) and right visual hemifield (E) in the control (black), left hemisphere (blue), and right hemisphere group (red). Asterisks indicate significant violations of the RMI (Bonferroni-corrected, p < .05). Significant positive RMI violations are indicative of MSI. Nonsignificant RMI violation values (less than or equal to 0) at all percentiles of the RT distribution indicate that the RTs in that audiovisual condition was fully explained by statistical facilitation (i.e., no MSI). Error bars indicate SEM.

Figure 2. 

The average of the median RTs in the A, V, and AV conditions in the left and right visual hemifield for each group (A, B, C). RMI violations in the left (D) and right visual hemifield (E) in the control (black), left hemisphere (blue), and right hemisphere group (red). Asterisks indicate significant violations of the RMI (Bonferroni-corrected, p < .05). Significant positive RMI violations are indicative of MSI. Nonsignificant RMI violation values (less than or equal to 0) at all percentiles of the RT distribution indicate that the RTs in that audiovisual condition was fully explained by statistical facilitation (i.e., no MSI). Error bars indicate SEM.

The lack of RMI violation in the left hemisphere group, as indicated by the one-sample t tests, could be the result of one patient with quite large negative RMI violation values relative to the other patients in the left hemisphere group (see Table 2, Patient 15). To explore this, we redid the one-sample t test analysis (to check for RMI violations) and the mixed ANOVA (to compare groups) excluding this patient from the left hemisphere group. The results were qualitatively similar and indicated reduced RMI violation in the left hemisphere group relative to the other two groups.

Table 2. 
Single-Case Analysis Results of Reductions in RMI Violation from the Control Group in the Left and Right Visual Hemifield for All 21 Patients
Patientn Deviating Quantiles Leftn Deviating Quantiles RightSum of Deviating Quantiles (Left + Right)Deviating RMI Violation?Deviating Mean RMI Violation?% rMRE Left% rMRE RightLesion LocationTotal Lesion Volume (ml)
No No 10 16 Left 68.48 
No No 18 17 Right 75.97 
No No −5 Right 107.02 
No No 10 20 Left 290.38 
No No 10 11 Right 58.08 
No No 27 22 Right 34.28 
Yes Yes 14 Left 73.06 
Yes Yes 16 Left 0.47 
No No 11 19 Right 6.08 
10 Yes Yes 18 −4 Left 82.79 
11 No No 23 18 Subcortical 2.09 
12 No No 14 Left 14.40 
13 No No 17 17 Left 61.69 
14 No No 10 15 Right 20.12 
15 18 Yes Yes −12 −7 Left 3.51 
16 No No 14 13 Right 46.90 
17 Yes Yes Left 0.34 
18 No No Left 186.46 
19 No No 29 17 Subcortical 2.08 
20 Yes No 13 Left 9.07 
21 12 Yes Yes −2 −4 Subcortical 6.42 
Patientn Deviating Quantiles Leftn Deviating Quantiles RightSum of Deviating Quantiles (Left + Right)Deviating RMI Violation?Deviating Mean RMI Violation?% rMRE Left% rMRE RightLesion LocationTotal Lesion Volume (ml)
No No 10 16 Left 68.48 
No No 18 17 Right 75.97 
No No −5 Right 107.02 
No No 10 20 Left 290.38 
No No 10 11 Right 58.08 
No No 27 22 Right 34.28 
Yes Yes 14 Left 73.06 
Yes Yes 16 Left 0.47 
No No 11 19 Right 6.08 
10 Yes Yes 18 −4 Left 82.79 
11 No No 23 18 Subcortical 2.09 
12 No No 14 Left 14.40 
13 No No 17 17 Left 61.69 
14 No No 10 15 Right 20.12 
15 18 Yes Yes −12 −7 Left 3.51 
16 No No 14 13 Right 46.90 
17 Yes Yes Left 0.34 
18 No No Left 186.46 
19 No No 29 17 Subcortical 2.08 
20 Yes No 13 Left 9.07 
21 12 Yes Yes −2 −4 Subcortical 6.42 

Additional RMI analyses for the left and right hemisphere lesion groups using a contralesional and ipsilesional RMI violation distinction revealed a similar pattern. Importantly, there was no difference in MSI between contralesional and ipsilesional targets. The details of these analyses can be found in the supplementary materials.3

MSI Impairments

The patient group consisted of patients with left-sided, right-sided, and subcortical brain damage in the brainstem and/or cerebellum. Depending on the lesion location, some patients may have been able to integrate auditory and visual information whereas others may not. This may have resulted in a (lack of) significant RMI violation at a group level when analyzing the left and right hemisphere lesion group. To more thoroughly investigate the impact of specific lesions on MSI, each patient's RMI violation in the left and right visual hemifield was compared with that of the control group using a two-tailed Crawford Howell's modified t test to test for deviations in RMI violation from the control group (i.e., within or outside the normal range). This was done to categorize patients as MSI impaired (reduced RMI violation) or not impaired and to subsequently (1) check whether there were still any benefits of multisensory stimulation in these two groups (i.e., rMRE), (2) how much reduction in rMRE an MSI impairment caused, and (3) to identify overlapping lesion locations using lesion subtraction of the two groups. An overview of the results of the single-case analysis is provided in Table 2 (RMI violations, lesion volume, deviating quantiles) and Figure 3 (lesion overlay and subtraction plots).

Figure 3. 

Lesion overlay and subtraction plots. The overlay plots show the number of patients with a lesion for a given voxel separately for patients with normal (A) and impaired (B) performance. The lesion subtraction plots (C) show which voxels are more frequently affected in patients with impaired performance compared with patients with normal performance by providing an absolute difference in lesion frequency between the groups. For example, the MSI impaired overlay plot shows that three of seven (43%) patients with MSI impairment have a lesion in the left putamen, whereas 3 of the 14 (21%) patients with no MSI impairment have a lesion in the left putamen. The lesion subtraction plot shows the resulting 22% difference in lesion prevalence. This finding suggests that the left putamen plays a role in MSI, but due to the qualitative nature of this analysis, it provides no statistical proof. The bar of the lesion subtraction plot ranges from 1% to 30% and is set to optimally visualize the regions that are more often affected in patients with MSI impairment than in patients with no MSI impairment (the maximum difference in lesion frequency was 36%).

Figure 3. 

Lesion overlay and subtraction plots. The overlay plots show the number of patients with a lesion for a given voxel separately for patients with normal (A) and impaired (B) performance. The lesion subtraction plots (C) show which voxels are more frequently affected in patients with impaired performance compared with patients with normal performance by providing an absolute difference in lesion frequency between the groups. For example, the MSI impaired overlay plot shows that three of seven (43%) patients with MSI impairment have a lesion in the left putamen, whereas 3 of the 14 (21%) patients with no MSI impairment have a lesion in the left putamen. The lesion subtraction plot shows the resulting 22% difference in lesion prevalence. This finding suggests that the left putamen plays a role in MSI, but due to the qualitative nature of this analysis, it provides no statistical proof. The bar of the lesion subtraction plot ranges from 1% to 30% and is set to optimally visualize the regions that are more often affected in patients with MSI impairment than in patients with no MSI impairment (the maximum difference in lesion frequency was 36%).

In total, 7 of 21 patients (33%) were labeled as MSI impaired in the left, right, or both visual hemifields. All but one patient with MSI impairment had a lesion in the left hemisphere (n = 6, 86%). This one patient had subcortical lesions in the brainstem and cerebellum. The severity of the reduction in RMI violation from the control group differed between patients as indicated by the variation in the number of deviating quantiles (ranging from 1 to 18 quantiles; see Table 2). Four patients had MSI impairments in both the left and right visual hemifield of which three had left hemisphere lesions and one had brainstem/cerebellar lesions. Two patients had MSI impairments only in the left visual hemifield (left hemisphere lesion), and one patient had MSI impairments only in the right visual hemifield (left hemisphere lesion). There was one patient (with a right hemisphere lesion) with increased RMI violation (a larger RMI violation value) relative to the control group at one quantile for targets presented in the left visual hemifield. This patient had no significant decrease in RMI violation (a lower RMI violation value) relative to the control group at any of the other quantiles (no MSI impairment) and was consequently labeled as not MSI impaired.

Relative MRE.

The impact of the MSI impairment on MRE (rMRE) was analyzed for the control (n = 14), no MSI impairment (n = 14), and MSI impairment (n = 7) groups. This was done to quantify how the benefit of multisensory stimulation changed due to an MSI impairment. That is, patients who did not integrate auditory and visual input could potentially still benefit from multisensory stimulation due to statistical facilitation (during independent processing of sensory input).

The control group (M = 15%, SE = 1.7) and the no MSI impairment group (M = 14%, SE = 1.8) showed significant rMRE (all t > 7, all ps < .001). The MSI impaired group did not show significant rMRE (M = 4.5%, SE = 3.0), t(6) = 1.493, p = .186. Thus, on average, the patient group with an impairment in MSI (as indicated by single-case analysis of RMI violations) did not benefit significantly from multisensory stimulation.

Lesion symptom analysis.

The lesion subtraction analysis revealed several brain regions that were related to MSI impairments. Lesions in the left caudate, left pallidum, left putamen, left thalamus, left insula, left postcentral and precentral gyrus, left central opercular cortex, left amygdala, and left frontal orbital cortex were more often damaged in patients with MSI impairment than in patients with no MSI impairment. These findings are in line with the literature as some of these areas are known to be involved in multisensory processing (Baum et al., 2012; Calvert, Hansen, Iversen, & Brammer, 2001; Calvert & Thesen, 2004; Crick & Koch, 2005; Remedios, Logothetis, & Kayser, 2010).

Unisensory processing.

A reduction in rMRE and reduced RMI violation could potentially be explained by differences in unisensory processing (e.g., see Van der Stoep et al., 2017; Otto, Dassy, & Mamassian, 2013). Although the group-level analyses of RTs (control vs. patient group) did not indicate any interaction between target modality (A, V, AV) and group, we investigated potential differences in unisensory RTs (A, V) between the MSI impairment, no MSI impairment, and control groups. A repeated-measures ANOVA with the factors Visual hemifield (left, right) and Group (control, MSI impairment, no MSI impairment) revealed no main effects of Visual hemifield or Group and no interaction between the two factors (0 < all Fs < 1.3, p > .1). On average, responses to auditory targets were 5 msec faster than to visual targets. These results provide further support that the impairments in MSI cannot simply be explained by differences in unisensory processing times between the three groups.

DISCUSSION

The integration of information from the different senses is an important function of the human brain as MSI improves detection, localization, and identification of events in the environment (Calvert et al., 2004). Studies of MSI in animal models and neuroimaging studies of MSI in humans have provided information about the neural mechanisms and substrates of MSI. However, human studies on the relation between specific brain areas and MSI impairments as measured in behavior in groups of patients with brain damage are scarce. Here, this relation was investigated by conducting a lesion analysis and relating brain damage to impairments in MSI. Critical areas for MSI were identified using lesion subtraction analysis.

Overall, both the control and patient groups showed MSI for targets in the left and right visual hemifield. The patient group was subsequently divided into patients with left and right hemisphere lesions to investigate the relation between side of the lesion, target presentation side (contralateral vs. ipsilateral), and the magnitude of MSI. The results indicated impaired MSI in both the contralesional and ipsilesional hemifield in the left hemisphere group, but not in the right hemisphere group. This impairment in MSI could not be explained by visual or auditory impairments, as there were no impairments in unisensory detection performance and no differences in unisensory RTs.

Given the heterogeneity of the lesions within the patient group, the amount of MSI in the left and right visual hemifield of each patient was compared with the healthy control group to determine impairments in MSI on an individual basis. Based on the results of these single-case analyses, patients were divided into a group with and a group without MSI impairment. In total, 33% of the patients in our sample demonstrated significant impairments in MSI after stroke. This percentage should be interpreted with care, because the sample was limited in size and biased. This bias resulted from the inclusion criteria for patients (chronic phase, able to perform the task, no neglect, etc.). Still, about one third of the patients had MSI impairments in the chronic phase, which suggests that MSI impairments may occur relatively frequently after stroke.

The lesion subtraction analysis indicated that the left caudate, left pallidum, left putamen, left thalamus, left insula, left postcentral and precentral gyrus, left central opercular cortex, left amygdala, and left frontal orbital cortex were more often damaged in patients with an impairment in MSI compared with patients without an impairment in MSI. The opercular cortex and parts of the frontal orbital postcentral and precentral gyrus surround the insular cortex. The opercular cortex partially overlaps with the superior temporal gyrus and the STS, which are known for containing neurons with multisensory response properties (e.g., Beauchamp, Argall, Bodurka, Duyn, & Martin, 2004; Beauchamp, 2005; Stevenson & James, 2009). These areas survived the 20% difference between groups criterion (see Figure 3C for other areas that were more often damaged in patients with MSI impairment regardless of the cutoff criterion) and have been shown to play a role in MSI (see Baum et al., 2012; Remedios et al., 2010; Crick & Koch, 2005; Calvert & Thesen, 2004; Calvert et al., 2001). Furthermore, these areas fit well within a subcortical tecto-thalamo-insula pathway, which has been proposed to play an important role in multisensory (temporal) processing (Bushara, Grafman, & Hallett, 2001; Calvert et al., 2001). In contrast with previous findings (see Ghazanfar & Schroeder, 2006), we did not find that areas in the partietal cortex (the intraparietal sulcus) were crucial for MSI. This may partly be due to the number of patients in our sample with lesions in potentially crucial areas like the ventral intraparietal area in our sample.

One patient with MSI impairment had brainstem and cerebellar lesions. The superior colliculi in the brainstem are well known for their multisensory responsive neurons (Calvert et al., 2001; Meredith & Stein, 1986). However, the lesions in this patient mainly covered the cerebellum and a small lower part of the brainstem, not including the superior colliculi. Less is known about the role of the cerebellum in multisensory perception. Recently, however, the cerebellum has been shown to have connections with the STS and seems to play a role in MSI as demonstrated by the lack of MSI after cerebellar agenesis (Ronconi et al., 2017; Sokolov et al., 2012).

Although voxelwise statistics were not possible, the current findings indicate that individuals with lesions in these left hemisphere brain areas may be especially susceptible to impairments in MSI (Calvert et al., 2000). Whereas neuroimaging studies have found activity related to MSI in both the left and right hemisphere (Beauchamp et al., 2004; Calvert et al., 2001), it has been observed that, specifically, the left insula and STS may be more responsive to multisensory stimuli than the corresponding areas in the right hemisphere (Calvert et al., 2001; though see Bushara et al., 2001, for contrasting findings regarding the right insula).

The lateralization of areas involved in MSI that was observed in the current study may, in part, be the result of the differing distribution of ischemic infarcts in the left and right hemisphere groups. This was indeed the case for the thalamus, which was damaged in 4 of 11 patients in the left hemisphere group and 0 patients in the right hemisphere group. However, the other brain areas that were identified by the lesion subtraction analysis were damaged in a similar number of patients in the left and right hemisphere groups, which makes this argument regarding lateralization less plausible for these areas. Regardless of the lateralization of MSI, the current findings indicate that, among others, lesions to the left insula and thalamus can cause an impairment in MSI in both the contralesional and/or ipsilesional visual hemifield. At a behavioral level, an MSI impairment resulted in a ∼70% reduction of MRE. Patients with MSI impairment did not benefit from multisensory stimulation when responding as fast as possible to the onset of audiovisual events.

These results are the first to clearly demonstrate that lesions to cortical and subcortical areas in the left hemisphere can result in MSI impairments as measured with a well-established behavioral paradigm in a group of patients with brain damage. More specifically, lesions to the left insula, claustrum, and striatum in the left hemisphere seem crucial for MSI impairments in both the contralesional and ipsilesional visual hemifield. The reduction in MRE in patients with left hemisphere lesions and MSI impairments was observed using a simple detection task in a quiet lab setting. Therefore, one could argue that these patients may also have problems in more complex and demanding situations in daily life. Unfortunately, impairments in MSI after stroke are generally not investigated in clinical settings. This may, in part, be the result of a lack of awareness of MSI deficits and the lack of simple and short tasks to measure MSI deficits in patients. MSI plays an important role in unifying information from different senses. Therefore, impairments in MSI may contribute to a feeling of being overstimulated by sensory stimuli, a complaint many stroke patients report subjectively. Two case studies of disruptions of MSI in the literature describe such problems in the temporal integration of auditory and visual input, causing the percept that vision and hearing are out of sync (Freeman et al., 2013; Hamilton et al., 2006). Patients P. H. (Freeman et al., 2013) and A. W. F. (Hamilton et al., 2006) both reported perceiving visual information with a delay relative to auditory information. A. W. F. reported that seeing lips when having a face-to-face conversation was so distracting that he started to look away from the face of the speaker to reduce interference. These experiences underline how impairments or alterations to multisensory processing can greatly influence daily life. The specific consequences of impairments in MSI in daily life remain unclear, however. Future studies of MSI in stroke patients in which problems in daily life are considered are required to reveal the importance and the plasticity of this basic but highly important sensory process.

The current study has several strengths. For example, the task in the current study and the RMI violation analysis have been widely used in the multisensory community to measure MSI (e.g., Van der Stoep et al., 2017; Otto & Mamassian, 2016; Van der Stoep, Van der Stigchel, et al., 2016; Van der Stoep, Van der Stigchel, et al., 2015; Girard et al., 2011; Ulrich et al., 2007; Diederich & Colonius, 2004; Gondan, Lange, Rösler, & Röder, 2004; Nozawa, Reuter-Lorenz, & Hughes, 1994; Miller, 1982, 1986, 1991; Raab, 1962). Previous studies have demonstrated that RMI violations relate to intracortical and extracortical measures of MSI (e.g., Molholm et al., 2002, 2006; Gondan et al., 2005), making the RMI measure an excellent proxy of MSI in a simple psychophysical task. Combining the results from a psychophysical task to measure MSI with lesion data provides a unique combination of methods to investigate MSI. A limitation of the current study is the fact that we did not have more information regarding the clinical outcome of the patients in our sample. Relating MSI impairments to neuropsychological assessments and questionnaires regarding experienced problems in daily life could have provided more information about the consequences of MSI impairments in daily life. In future studies, it could also be relevant to investigate the brain networks that are involved in MSI, given that lesions to different brain regions within the same network could result in similar problems.

To conclude, the current study has provided unique insights into MSI by investigating the link between brain damage and behavioral measures of MSI. The results indicate that MSI can be specifically impaired after stroke (in 33% of the patients in our sample) and that lesions to the critical areas in the left hemisphere (left nucleus caudatus, left globus pallidum, left putamen, left thalamus, left insular cortex, and left postcentral and precentral gyrus) cause clear MSI impairments. The current findings also highlight the importance of the ability to integrate auditory and visual input given that MSI impairments led to a ∼70% reduction in MRE in both the left and right visual hemifield.

Acknowledgments

This study was supported by two grants from the Netherlands Organization for Scientific Research (grant 451-17-014 to N. V. D. S. and grant 451-10-013 to T. C. W. N.). The authors would like to thank Dorien Slabbers, Dr. G. J. Biessels, Dr. L. J. Kappelle, and Dr. G. J. E. Rinkel from the Department of Neurology and Neurosurgery of the University Medical Center Utrecht, the Netherlands, for access to the stroke registry for the purpose of this study.

Reprint requests should be sent to Nathan Van der Stoep, Department of Experimental Psychology, Helmholtz Insitute, Utrecht University, Heidelberglaan 1, 3584 CS, Utrecht, The Netherlands, or via e-mail: N.VanderStoep@uu.nl.

Notes

1. 

Superadditive responses are, perhaps, most well known but are one of several types of multisensory interactions that have been observed in multisensory neurons. Additive, subadditive, and other modulatory responses have also been observed (see, e.g., Stein & Stanford, 2008; Meredith & Stein, 1986, for more on the type of interactions observed in multisensory neurons).

2. 

These cutoff criteria were determined based on the line bisection error in the neurotypical control group averaged across all nine lines ±2.5 SDs. See Van der Stoep et al. (2013) for a description of the line bisection task that was used in near space.

3. 

Supplementary materials for this paper can be retrieved from https://surfdrive.surf.nl/files/index.php/s/Xx3zqxO0DUqQGj2.

REFERENCES

Alais
,
D.
, &
Burr
,
D.
(
2004
).
The ventriloquist effect results from near-optimal bimodal integration
.
Current Biology
,
14
,
257
262
.
Bamford
,
J.
,
Sandercock
,
P.
,
Dennis
,
M.
,
Burn
,
J.
, &
Warlow
,
C.
(
1991
).
Classification and natural history of clinically identifiable subtypes of cerebral infarction
.
Lancet
,
337
,
1521
1526
.
Baum
,
S. H.
,
Martin
,
R. C.
,
Hamilton
,
A. C.
, &
Beauchamp
,
M. S.
(
2012
).
Multisensory speech perception without the left superior temporal sulcus
.
Neuroimage
,
62
,
1825
1832
.
Beauchamp
,
M. S.
(
2005
).
See me, hear me, touch me: Multisensory integration in lateral occipital-temporal cortex
.
Current Opinion in Neurobiology
,
15
,
145
153
.
Beauchamp
,
M. S.
,
Argall
,
B. D.
,
Bodurka
,
J.
,
Duyn
,
J. H.
, &
Martin
,
A.
(
2004
).
Unraveling multisensory integration: Patchy organization within human STS multisensory cortex
.
Nature Neuroscience
,
7
,
1190
1192
.
Beauchamp
,
M. S.
,
Nath
,
A. R.
, &
Pasalar
,
S.
(
2010
).
fMRI-guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect
.
Journal of Neuroscience
,
30
,
2414
2417
.
Biesbroek
,
J. M.
,
Weaver
,
N. A.
, &
Biessels
,
G. J.
(
2017
).
Lesion location and cognitive impact of cerebral small vessel disease
.
Clinical Science
,
131
,
715
728
.
Brainard
,
D. H.
(
1997
).
The psychophysics toolbox
.
Spatial Vision
,
10
,
433
436
.
Bushara
,
K. O.
,
Grafman
,
J.
, &
Hallett
,
M.
(
2001
).
Neural correlates of auditory-visual stimulus onset asynchrony detection
.
Journal of Neuroscience
,
21
,
300
304
.
Calvert
,
G. A.
,
Campbell
,
R.
, &
Brammer
,
M. J.
(
2000
).
Evidence from functional magnetic resonance imaging of crossmodal binding in the human heteromodal cortex
.
Current Biology
,
10
,
649
657
.
Calvert
,
G. A.
,
Hansen
,
P. C.
,
Iversen
,
S. D.
, &
Brammer
,
M. J.
(
2001
).
Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect
.
Neuroimage
,
14
,
427
438
.
Calvert
,
G. A.
,
Spence
,
C.
, &
Stein
,
B. E.
(
2004
).
The handbook of multisensory processes
.
Cambridge, MA
:
MIT Press
.
Calvert
,
G. A.
, &
Thesen
,
T.
(
2004
).
Multisensory integration: Methodological approaches and emerging principles in the human brain
.
Journal of Physiology, Paris
,
98
,
191
205
.
Cappe
,
C.
,
Thut
,
G.
,
Romei
,
V.
, &
Murray
,
M. M.
(
2010
).
Auditory-visual multisensory interactions in humans: Timing, topography, directionality, and sources
.
Journal of Neuroscience
,
30
,
12572
12580
.
Cecere
,
R.
,
Romei
,
V.
,
Bertini
,
C.
, &
Làdavas
,
E.
(
2014
).
Crossmodal enhancement of visual orientation discrimination by looming sounds requires functional activation of primary visual areas: A case study
.
Neuropsychologia
,
56
,
350
358
.
Crawford
,
J. R.
, &
Garthwaite
,
P. H.
(
2005
).
Testing for suspected impairments and dissociations in single-case studies in neuropsychology: Evaluation of alternatives using monte carlo simulations and revised tests for dissociations
.
Neuropsychology
,
19
,
318
331
.
Crick
,
F. C.
, &
Koch
,
C.
(
2005
).
What is the function of the claustrum?
Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences
,
360
,
1271
1279
.
Desikan
,
R. S.
,
Ségonne
,
F.
,
Fischl
,
B.
,
Quinn
,
B. T.
,
Dickerson
,
B. C.
,
Blacker
,
D.
, et al
(
2006
).
An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest
.
Neuroimage
,
31
,
968
980
.
Diederich
,
A.
, &
Colonius
,
H.
(
2004
).
Bimodal and trimodal multisensory enhancement: Effects of stimulus onset and intensity on reaction time
.
Perception & Psychophysics
,
66
,
1388
1404
.
Ernst
,
M. O.
, &
Banks
,
M. S.
(
2002
).
Humans integrate visual and haptic information in a statistically optimal fashion
.
Nature
,
415
,
429
433
.
Farnè
,
A.
, &
Làdavas
,
E.
(
2002
).
Auditory peripersonal space in humans
.
Journal of Cognitive Neuroscience
,
14
,
1030
1043
.
Fiebelkorn
,
I. C.
,
Foxe
,
J. J.
,
Butler
,
J. S.
, &
Molholm
,
S.
(
2011
).
Auditory facilitation of visual-target detection persists regardless of retinal eccentricity and despite wide audiovisual misalignments
.
Experimental Brain Research
,
213
,
167
174
.
Fogassi
,
L.
,
Gallese
,
V.
,
Fadiga
,
L.
,
Luppino
,
G.
,
Matelli
,
M.
, &
Rizzolatti
,
G.
(
1996
).
Coding of peripersonal space in inferior premotor cortex (area F4)
.
Journal of Neurophysiology
,
76
,
141
157
.
Fonov
,
V. S.
,
Evans
,
A. C.
,
McKinstry
,
R. C.
,
Almli
,
C. R.
, &
Collins
,
D. L.
(
2009
).
Unbiased nonlinear average age-appropriate brain templates from birth to adulthood
.
Neuroimage
,
47
,
S102
.
Frassinetti
,
F.
,
Bolognini
,
N.
,
Bottari
,
D.
,
Bonora
,
A.
, &
Làdavas
,
E.
(
2005
).
Audiovisual integration in patients with visual deficit
.
Journal of Cognitive Neuroscience
,
17
,
1442
1452
.
Freeman
,
E. D.
,
Ipser
,
A.
,
Palmbaha
,
A.
,
Paunoiu
,
D.
,
Brown
,
P.
,
Lambert
,
C.
, et al
(
2013
).
Sight and sound out of synch: Fragmentation and renormalisation of audiovisual integration and subjective timing
.
Cortex
,
49
,
2875
2887
.
Ghazanfar
,
A. A.
, &
Schroeder
,
C. E.
(
2006
).
Is neocortex essentially multisensory?
Trends in Cognitive Sciences
,
10
,
278
285
.
Girard
,
S.
,
Collignon
,
O.
, &
Lepore
,
F.
(
2011
).
Multisensory gain within and across hemispaces in simple and choice reaction time paradigms
.
Experimental Brain Research
,
214
,
1
8
.
Girard
,
S.
,
Pelland
,
M.
,
Lepore
,
F.
, &
Collignon
,
O.
(
2013
).
Impact of the spatial congruence of redundant targets on within-modal and cross-modal integration
.
Experimental Brain Research
,
224
,
275
285
.
Gondan
,
M.
,
Lange
,
K.
,
Rösler
,
F.
, &
Röder
,
B.
(
2004
).
The redundant target effect is affected by modality switch costs
.
Psychonomic Bulletin & Review
,
11
,
307
313
.
Gondan
,
M.
, &
Minakata
,
K.
(
2016
).
A tutorial on testing the race model inequality
.
Attention, Perception & Psychophysics
,
78
,
723
735
.
Gondan
,
M.
,
Niederhaus
,
B.
,
Rösler
,
F.
, &
Röder
,
B.
(
2005
).
Multisensory processing in the redundant-target effect: A behavioral and event-related potential study
.
Perception & Psychophysics
,
67
,
713
726
.
Graziano
,
M. S.
(
1994
).
The representation of extrapersonal space: A possible role for bimodal, visual-tactile neurons
. In
M.
Gazzaniga
(Ed.),
C. G.
Gross
(Trans.),
The cognitive neurosciences
(pp.
1021
1034
).
Cambridge, MA
:
MIT Press
.
Graziano
,
M. S.
, &
Gross
,
C. G.
(
1998
).
Spatial maps for the control of movement
.
Current Opinion in Neurobiology
,
8
,
195
201
.
Gross
,
C. G.
, &
Graziano
,
M. S. A.
(
1995
).
Multiple representations of space in the brain
.
The Neuroscientist
,
1
,
43
50
.
Hamilton
,
R. H.
,
Shenton
,
J. T.
, &
Coslett
,
H. B.
(
2006
).
An acquired deficit of audiovisual speech processing
.
Brain and Language
,
98
,
66
73
.
Hughes
,
H. C.
,
Reuter-Lorenz
,
P. A.
,
Nozawa
,
G.
, &
Fendrich
,
R.
(
1994
).
Visual-auditory interactions in sensorimotor processing: Saccades versus manual responses
.
Journal of Experimental Psychology: Human Perception and Performance
,
20
,
131
153
.
Klein
,
S.
,
Staring
,
M.
,
Murphy
,
K.
,
Viergever
,
M. A.
, &
Pluim
,
J. P. W.
(
2010
).
elastix: A toolbox for intensity-based medical image registration
.
IEEE Transactions on Medical Imaging
,
29
,
196
205
.
Làdavas
,
E.
(
2008
).
Multisensory-based approach to the recovery of unisensory deficit
.
Annals of the New York Academy of Sciences
,
1124
,
98
110
.
Làdavas
,
E.
, &
Farnè
,
A.
(
2004
).
Visuo-tactile representation of near-the-body space
.
Journal of Physiology, Paris
,
98
,
161
170
.
Leone
,
L. M.
, &
McCourt
,
M. E.
(
2013
).
The roles of physical and physiological simultaneity in audiovisual multisensory facilitation
.
I-Perception
,
4
,
213
228
.
Lovelace
,
C. T.
,
Stein
,
B. E.
, &
Wallace
,
M. T.
(
2003
).
An irrelevant light enhances auditory detection in humans: A psychophysical analysis of multisensory integration in stimulus detection
.
Brain Research, Cognitive Brain Research
,
17
,
447
453
.
Martuzzi
,
R.
,
Murray
,
M. M.
,
Michel
,
C. M.
,
Thiran
,
J. P.
,
Maeder
,
P. P.
,
Clarke
,
S.
, et al
(
2007
).
Multisensory interactions within human primary cortices revealed by BOLD dynamics
.
Cerebral Cortex
,
17
,
1672
1679
.
Meredith
,
M. A.
,
Nemitz
,
J. W.
, &
Stein
,
B. E.
(
1987
).
Determinants of multisensory integration in superior colliculus neurons. I. Temporal factors
.
Journal of Neuroscience
,
7
,
3215
3229
.
Meredith
,
M. A.
, &
Stein
,
B. E.
(
1986
).
Spatial factors determine the activity of multisensory neurons in cat superior colliculus
.
Brain Research
,
365
,
350
354
.
Meredith
,
M. A.
, &
Stein
,
B. E.
(
1996
).
Spatial determinants of multisensory integration in cat superior colliculus neurons
.
Journal of Neurophysiology
,
75
,
1843
1857
.
Miller
,
J.
(
1982
).
Divided attention: Evidence for coactivation with redundant signals
.
Cognitive Psychology
,
14
,
247
279
.
Miller
,
J.
(
1986
).
Timecourse of coactivation in bimodal divided attention
.
Perception & Psychophysics
,
40
,
331
343
.
Miller
,
J.
(
1991
).
Channel interaction and the redundant-targets effect in bimodal divided attention
.
Journal of Experimental Psychology: Human Perception and Performance
,
17
,
160
169
.
Miller
,
J.
(
2016
).
Statistical facilitation and the redundant signals effect: What are race and coactivation models?
Attention, Perception & Psychophysics
,
78
,
516
519
.
Molholm
,
S.
,
Ritter
,
W.
,
Murray
,
M. M.
,
Javitt
,
D. C.
,
Schroeder
,
C. E.
, &
Foxe
,
J. J.
(
2002
).
Multisensory auditory-visual interactions during early sensory processing in humans: A high-density electrical mapping study
.
Brain Research, Cognitive Brain Research
,
14
,
115
128
.
Molholm
,
S.
,
Sehatpour
,
P.
,
Mehta
,
A. D.
,
Shpaner
,
M.
,
Gomez-Ramirez
,
M.
,
Ortigue
,
S.
, et al
(
2006
).
Audio-visual multisensory integration in superior parietal lobule revealed by human intracranial recordings
.
Journal of Neurophysiology
,
96
,
721
729
.
Murray
,
M. M.
,
Cappe
,
C.
,
Romei
,
V.
,
Martuzzi
,
R.
, &
Thut
,
G.
(
2012
).
Auditory-visual multisensory interactions in human primary cortices: Synthesis and controversies
. In
B. E.
Stein
(Ed.),
The new handbook of multisensory processes
(pp.
223
238
).
Cambridge, MA
:
MIT Press
.
Nardini
,
M.
,
Bales
,
J.
, &
Mareschal
,
D.
(
2015
).
Integration of audio-visual information for spatial decisions in children and adults
.
Developmental Science
,
19
,
803
816
.
Nath
,
A. R.
, &
Beauchamp
,
M. S.
(
2011
).
Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech
.
Journal of Neuroscience
,
31
,
1704
1714
.
Nozawa
,
G.
,
Reuter-Lorenz
,
P. A.
, &
Hughes
,
H. C.
(
1994
).
Parallel and serial processes in the human oculomotor system: Bimodal integration and express saccades
.
Biological Cybernetics
,
72
,
19
34
.
Oliveira
,
C. B.
,
Medeiros
,
Í. R.
,
Greters
,
M. G.
,
Frota
,
N. A.
,
Lucato
,
L. T.
,
Scaff
,
M.
, et al
(
2011
).
Abnormal sensory integration affects balance control in hemiparetic patients within the first year after stroke
.
Clinics
,
66
,
2043
2048
.
Otto
,
T. U.
,
Dassy
,
B.
, &
Mamassian
,
P.
(
2013
).
Principles of multisensory behavior
.
Journal of Neuroscience
,
33
,
7463
7474
.
Otto
,
T. U.
, &
Mamassian
,
P.
(
2016
).
Multisensory decisions: The test of a race model, its logic, and power
.
Multisensory Research
,
30
,
1
24
.
Passamonti
,
C.
,
Bertini
,
C.
, &
Làdavas
,
E.
(
2009
).
Audio-visual stimulation improves oculomotor patterns in patients with hemianopia
.
Neuropsychologia
,
47
,
546
555
.
Raab
,
D. H.
(
1962
).
Statistical facilitation of simple reaction times
.
Transactions of the New York Academy of Sciences
,
24
,
574
590
.
Remedios
,
R.
,
Logothetis
,
N. K.
, &
Kayser
,
C.
(
2010
).
Unimodal responses prevail within the multisensory claustrum
.
Journal of Neuroscience
,
30
,
12902
12907
.
Ronconi
,
L.
,
Casartelli
,
L.
,
Carna
,
S.
,
Molteni
,
M.
,
Arrigoni
,
F.
, &
Borgatti
,
R.
(
2017
).
When one is enough: Impaired multisensory integration in cerebellar agenesis
.
Cerebral Cortex
,
27
,
2041
2051
.
Rorden
,
C.
,
Karnath
,
H. O.
, &
Bonilha
,
L.
(
2007
).
Improving lesion-symptom mapping
.
Journal of Cognitive Neuroscience
,
19
,
1081
1088
.
Ross
,
L. A.
,
Saint-Amour
,
D.
,
Leavitt
,
V. M.
,
Javitt
,
D. C.
, &
Foxe
,
J. J.
(
2007
).
Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments
.
Cerebral Cortex
,
17
,
1147
1153
.
Senkowski
,
D.
,
Saint-Amour
,
D.
,
Höfle
,
M.
, &
Foxe
,
J. J.
(
2011
).
Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness
.
Neuroimage
,
56
,
2200
2208
.
Serino
,
A.
,
Canzoneri
,
E.
, &
Avenanti
,
A.
(
2011
).
Fronto-parietal areas necessary for a multisensory representation of peripersonal space in humans: An rTMS study
.
Journal of Cognitive Neuroscience
,
23
,
2956
2967
.
Serino
,
A.
,
Noel
,
J.-P.
,
Galli
,
G.
,
Canzoneri
,
E.
,
Marmaroli
,
P.
,
Lissek
,
H.
, et al
(
2015
).
Body part-centered and full body-centered peripersonal space representations
.
Scientific Reports
,
5
,
18603
.
Sokolov
,
A. A.
,
Erb
,
M.
,
Gharabaghi
,
A.
,
Grodd
,
W.
,
Tatagiba
,
M. S.
, &
Pavlova
,
M. A.
(
2012
).
Biological motion processing: The left cerebellum communicates with the right superior temporal sulcus
.
Neuroimage
,
59
,
2824
2830
.
Spence
,
C.
, &
Driver
,
J.
(Eds.). (
2004
).
Crossmodal space and crossmodal attention
(1st ed.).
Oxford
:
Oxford University Press
.
Stein
,
B. E.
, &
Meredith
,
A.
(
1993
).
The merging of the senses
(1st ed.).
Cambridge, MA
:
MIT Press
.
Stein
,
B. E.
, &
Stanford
,
T. R.
(
2008
).
Multisensory integration: Current issues from the perspective of the single neuron
.
Nature Reviews Neuroscience
,
9
,
255
266
.
Stevenson
,
R. A.
,
Ghose
,
D.
,
Fister
,
J. K.
,
Sarko
,
D. K.
,
Altieri
,
N. A.
,
Nidiffer
,
A. R.
, et al
(
2014
).
Identifying and quantifying multisensory integration: A tutorial review
.
Brain Topography
,
27
,
707
730
.
Stevenson
,
R. A.
, &
James
,
T. W.
(
2009
).
Audiovisual integration in human superior temporal sulcus: Inverse effectiveness and the neural processing of speech and object recognition
.
Neuroimage
,
44
,
1210
1223
.
Talsma
,
D.
,
Doty
,
T. J.
, &
Woldorff
,
M. G.
(
2007
).
Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration?
Cerebral Cortex
,
17
,
679
690
.
Ten Brink
,
A. F.
,
Verwer
,
J. H.
,
Biesbroek
,
J. M.
,
Visser-Meily
,
J. M. A.
, &
Nijboer
,
T. C. W.
(
2017
).
Differences between left- and right-sided neglect revisited: A large cohort study across multiple domains
.
Journal of Clinical and Experimental Neuropsychology
,
39
,
707
723
.
Tinga
,
A. M.
,
Visser-Meily
,
J. M.
,
van der Smagt
,
M. J.
,
Van der Stigchel
,
S.
,
van Ee
,
R.
, &
Nijboer
,
T. C. W.
(
2016
).
Multisensory stimulation to improve low- and higher-level sensory deficits after stroke: A systematic review
.
Neuropsychology Review
,
26
,
73
91
.
Tyll
,
S.
,
Bonath
,
B.
,
Schoenfeld
,
M. A.
,
Heinze
,
H. J.
,
Ohl
,
F. W.
, &
Noesselt
,
T.
(
2013
).
Neural basis of multisensory looming signals
.
Neuroimage
,
65
,
13
22
.
Ulrich
,
R.
,
Miller
,
J.
, &
Schröter
,
H.
(
2007
).
Testing the race model inequality: An algorithm and computer programs
.
Behavior Research Methods
,
39
,
291
302
.
Van der Stoep
,
N.
,
Nijboer
,
T. C.
,
Van der Stigchel
,
S.
, &
Spence
,
C.
(
2015
).
Multisensory interactions in the depth plane in front and rear space: A review
.
Neuropsychologia
,
70
,
335
349
.
Van der Stoep
,
N.
,
Serino
,
A.
,
Farnè
,
A.
,
Di Luca
,
M.
, &
Spence
,
C.
(
2016
).
Depth: The forgotten dimension in multisensory research
.
Multisensory Research
,
29
,
493
524
.
Van der Stoep
,
N.
,
Spence
,
C.
,
Nijboer
,
T. C. W.
, &
Van der Stigchel
,
S.
(
2015
).
On the relative contributions of multisensory integration and crossmodal exogenous spatial attention to multisensory response enhancement
.
Acta Psychologica
,
162
,
20
28
.
Van der Stoep
,
N.
,
Van der Stigchel
,
S.
, &
Nijboer
,
T. C. W.
(
2015
).
Exogenous spatial attention decreases audiovisual integration
.
Attention, Perception & Psychophysics
,
77
,
464
482
.
Van der Stoep
,
N.
,
Van der Stigchel
,
S.
,
Nijboer
,
T. C. W.
, &
Spence
,
C.
(
2017
).
Visually induced inhibition of return affects the integration of auditory and visual information
.
Perception
,
46
,
6
17
.
Van der Stoep
,
N.
,
Van der Stigchel
,
S.
,
Nijboer
,
T. C. W.
, &
Van der Smagt
,
M. J.
(
2016
).
Audiovisual integration in near and far space: Effects of changes in distance and stimulus effectiveness
.
Experimental Brain Research
,
234
,
1175
1188
.
Van der Stoep
,
N.
,
Visser-Meily
,
J. M.
,
Kappelle
,
L. J.
,
de Kort
,
P. L.
,
Huisman
,
K. D.
,
Eijsackers
,
A. L.
, et al
(
2013
).
Exploring near and far regions of space: Distance-specific visuospatial neglect after stroke
.
Journal of Clinical and Experimental Neuropsychology
,
35
,
799
811
.
Van Strien
,
J.
(
2003
).
The Dutch handedness questionnaire
. .
Wallace
,
M. T.
,
Meredith
,
M. A.
, &
Stein
,
B. E.
(
1998
).
Multisensory integration in the superior colliculus of the alert cat
.
Journal of Neurophysiology
,
80
,
1006
1010
.
Whelan
,
R.
(
2008
).
Effective analysis of reaction time data
.
The Psychological Record
,
58
,
475
482
.
Yau
,
J. M.
,
DeAngelis
,
G. C.
, &
Angelaki
,
D. E.
(
2015
).
Dissecting neural circuits for multisensory integration and crossmodal processing
.
Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences
,
370
,
20140203
.
Zhao
,
L.
,
Biesbroek
,
J. M.
,
Shi
,
L.
,
Liu
,
W.
,
Kuijf
,
H. J.
,
Chu
,
W. W.
, et al
(
2017
).
Strategic infarct location for post-stroke cognitive impairment: A multivariate lesion-symptom mapping study
.
Journal of Cerebral Blood Flow and Metabolism
,
38
,
1299
1311
.