Abstract

Multisensory stimuli can improve performance, facilitating RTs on sensorimotor tasks. This benefit is referred to as the redundant signals effect (RSE) and can exceed predictions on the basis of probability summation, indicative of integrative processes. Although an RSE exceeding probability summation has been repeatedly observed in humans and nonprimate animals, there are scant and inconsistent data from nonhuman primates performing similar protocols. Rather, existing paradigms have instead focused on saccadic eye movements. Moreover, the extant results in monkeys leave unresolved how stimulus synchronicity and intensity impact performance. Two trained monkeys performed a simple detection task involving arm movements to auditory, visual, or synchronous auditory–visual multisensory pairs. RSEs in excess of predictions on the basis of probability summation were observed and thus forcibly follow from neural response interactions. Parametric variation of auditory stimulus intensity revealed that in both animals, RT facilitation was limited to situations where the auditory stimulus intensity was below or up to 20 dB above perceptual threshold, despite the visual stimulus always being suprathreshold. No RT facilitation or even behavioral costs were obtained with auditory intensities 30–40 dB above threshold. The present study demonstrates the feasibility and the suitability of behaving monkeys for investigating links between psychophysical and neurophysiologic instantiations of multisensory interactions.

INTRODUCTION

The environment provides multisensory inputs to an organism. Because the combination of different sensory inputs can reduce perceptual ambiguity and enhance the speed and accuracy of stimulus detection (e.g., Cappe, Thut, Romei, & Murray, 2009; Romei, Murray, Cappe, & Thut, 2009; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Stein, London, Wilkinson, & Price, 1996; Sumby & Pollack, 1954), the ability to integrate cues across multiple senses is likely advantageous for survival. Across species ranging from single-cell organisms to humans, it has been shown that behavior with multisensory stimuli is quicker and/or more accurate than that to their unisensory components (Stein & Meredith, 1993; Welch & Warren, 1986). This behavioral facilitation under multisensory conditions is one instantiation of the so-called redundant signals effect (RSE; Nickerson, 1973; Raab, 1962; Todd, 1912). The extent to which nonhuman primates, specifically macaque monkeys, exhibit an RSE in their manual RTs (RTs) to auditory–visual multisensory stimuli is unresolved and was the focus of the present study. Prior research in monkeys observed an RSE only for manual RTs when stimuli were presented asynchronously, although it is noteworthy that such was not statistically validated (Miller, Ulrich, & Lamarre, 2001; Lamarre, Busby, & Spidalieri, 1983). By contrast, studies in humans have successfully used synchronously presented stimuli to elicit an RSE (e.g., Cappe, Thut, et al., 2009; Romei et al., 2009; Martuzzi et al., 2007; Romei, Murray, Merabet, & Thut, 2007; Molholm et al., 2002; Giard & Peronnet, 1999). Determining the conditions under which an RSE is observable in monkeys is an essential step for ultimately drawing (causal) links between neurophysiologic and behavioral indices of multisensory processes not only in animals (e.g., Wang, Celebrini, Trotter, & Barone, 2008) but also across species (for results in humans, see Romei et al., 2007, 2009; Sperdin, Cappe, Foxe, & Murray, 2009).

To date, studies of the impact of multisensory stimulation on monkey behavior have focused either on saccadic eye movements to visual or auditory–visual targets (Wang et al., 2008; Frens & Van Opstal, 1998) or on preferential looking paradigms that required the monkey to use auditory information as a basis for which visual stream to look at (Maier, Neuhoff, Logothetis, & Ghazanfar, 2004; Ghazanfar & Logothetis, 2003; for forced-choice data from a chimpanzee, see also Izumi & Kojima, 2004). One downside with these paradigms is that they cannot exclude the role of selective attention to one or the other sensory modality, nor can they provide a metric of truly integrative processes, because the requisite unisensory conditions were not included. This being said, eye movements are a quintessential feature for the perception of naturalistic stimuli, and monkeys and humans have stereotypical viewing patterns when looking at dynamic faces (Ghazanfar, Nielsen, & Logothetis, 2006; Vatikiotis-Bateson, Eigsti, Yano, & Munhall, 1998). However, given the known impact of eye position on neural responsiveness (e.g., Bell, Meredith, Van Opstal, & Munoz, 2005; Werner-Reiss, Kelly, Trause, Underhill, & Groh, 2003; Corneil, Van Wanrooij, Munoz, & Van Opstal, 2002; for a contrary example when recording from area VIP, see Schlack, Sterbing-D'Angelo, Hartung, Hoffmann, & Bremmer, 2005), electrophysiological and brain imaging studies in awake animals typically required central fixation and no other behavior (e.g., Kayser & Logothetis, 2009; Kayser, Petkov, & Logothetis, 2008; Ghazanfar, Maier, Hoffman, & Logothetis, 2005; Bell, Corneil, Meredith, & Munoz, 2001). Plus, fixation is nearly universally required in human brain imaging (fMRI, PET, EEG, and MEG). Importantly, as far as we know, fixation tasks have no influence at a behavioral level (e.g., Hughes, Reuter-Lorenz, Nozawa, & Fendrich, 1994). Moreover, we sought to have the animals fixate here because most electrophysiological studies are conducted while the animal fixates to avoid confounds from muscle activity and to minimize uncertainty about task compliance and the kind of visual input entering the neuron's receptive field on a given trial.

More generally, there is a gap between our understanding of neural manifestations of multisensory interactions and their causal relationship with behavior. Similarly, the extant behavioral results leave it unclear whether monkeys will exhibit RSEs in response to auditory–visual multisensory stimuli akin to those frequently observed in humans (e.g., Cappe, Thut, et al., 2009; Martuzzi et al., 2007; Romei et al., 2007; Molholm et al., 2002; Giard & Peronnet, 1999). Establishing such would provide a model system that can in turn be the focus of neurophysiological and neuropharmacological research, particularly given the increased evidence for a role of multisensory phenomena in development (Neil, Chee-Ruiter, Scheier, Lewkowicz, & Shimojo, 2006; Wallace, Carriere, Perrault, Vaughan, & Stein, 2006), aging (Laurienti, Burdette, Maldjian, & Wallace, 2006), and clinical syndromes (e.g., Blau, van Atteveldt, Ekkebus, Goebel, & Blomert, 2009; Ross, Saint-Amour, Leavitt, Molholm, et al., 2007; Hairston, Burdette, Flowers, Wood, & Wallace, 2005). Consequently, the principal objective of this study was to use performance on a manual RT task as an index of multisensory processes in trained macaque monkeys.

In the present study, we used a simple detection task, in which a reaching arm movement was triggered by presentation of auditory (A), visual (V), or synchronous multisensory auditory–visual (AV) stimuli. Analyses of accuracy rates and RTs allowed us to determine if monkeys exhibit an RSE with similar gains to those typically observed in human observers. The assessment of whether RT facilitation exceeded levels predicted by probability summation allowed us to determine whether integrative processes need to be invoked to account for behavioral improvements. Given the pattern of results obtained by Miller et al. (2001) and Lamarre et al. (1983), who failed to observe an RSE with synchronously presented and suprathreshold auditory and visual stimuli, and given the fact that studies in humans routinely obtain an RSE with synchronously presented auditory–visual stimuli, we parametrically varied the efficacy of auditory stimuli. We hypothesized that multisensory stimulus presentation facilitates detection near threshold and shortens RT as compared with unisensory stimulus presentation (cf. Stein & Meredith, 1993). The assumption is that unisensory stimuli, which may be difficult to perceive near threshold, may be detected with higher probability and faster (shorter RT) when delivered together as a multisensory stimulus pair. To this end, the threshold of an auditory stimulus was determined for two monkeys. Then, accuracy rates and RTs were determined at different acoustic intensity levels and compared with multisensory conditions when the same auditory stimulus was presented simultaneously with a suprathreshold visual stimulus.

METHODS

Behavioral Task

Two adult male macaque monkeys (Macaca fascicularis; MK1 and MK2) were trained to perform an AV detection task (Figure 1) that required an arm-reaching movement in response to auditory (A), visual (V), or simultaneous multisensory auditory–visual (AV) stimuli. Data included in the present analyses were collected over a period when both monkeys were aged between 3 and 4 years. All experimental protocols, including care, surgery, and training of animals, were performed according to the Guide for the Care and Use of Laboratory Animals (ISBN 0-309-05377-3, 1996) and were approved by local (Swiss) veterinary authorities.

Figure 1. 

Schematic representation of a typical trial of the task. (A) Schematic representation of the experimental setup. The monkey sat in its primate chair and faced a vertical transparent glass in front of a computer monitor. Below the monitor, a horizontal touch-sensitive pad (bottom, gray rectangle) was used as start lever to initiate a trial. The monkey had to fixate the cross displayed on the monitor at the same time and throughout the duration of the trial. Next to the start lever, a pellet basin was used to deliver the reward (not shown). The transparent glass was adorned with a panel composed of a key detector (black square), representing the pad target to be touched by the monkey after presentation of the stimulus. (B) Schematic representation of the temporal sequence of events during a typical trial. Row 1 indicates the position of the different events imposed on the monkey. Row 2 illustrates the monkey's motor behavior. From trial initiation, the monkey maintained its left hand on the start lever until the stimulus was delivered, followed by an RT preceding the onset of the pointing movement to the target (MT = motor time, corresponding to the time for the hand to reach the pad target).

Figure 1. 

Schematic representation of a typical trial of the task. (A) Schematic representation of the experimental setup. The monkey sat in its primate chair and faced a vertical transparent glass in front of a computer monitor. Below the monitor, a horizontal touch-sensitive pad (bottom, gray rectangle) was used as start lever to initiate a trial. The monkey had to fixate the cross displayed on the monitor at the same time and throughout the duration of the trial. Next to the start lever, a pellet basin was used to deliver the reward (not shown). The transparent glass was adorned with a panel composed of a key detector (black square), representing the pad target to be touched by the monkey after presentation of the stimulus. (B) Schematic representation of the temporal sequence of events during a typical trial. Row 1 indicates the position of the different events imposed on the monkey. Row 2 illustrates the monkey's motor behavior. From trial initiation, the monkey maintained its left hand on the start lever until the stimulus was delivered, followed by an RT preceding the onset of the pointing movement to the target (MT = motor time, corresponding to the time for the hand to reach the pad target).

All training and recording sessions took place in an electrically shielded, sound-attenuated chamber (see Durif, Jouffrais, & Rouiller, 2003). Using pellets as reward, the animal was trained to maintain fixation on a cross located at the center of the computer monitor facing the animal. Eye position was monitored using an ISCAN eye-tracking system. To initiate a trial, the monkey had to place the left hand on a starting lever, prompting the appearance of the fixation point on the monitor. Then, the monkey had to fixate during the entire trial. If the monkey did not maintain his fixation, the trial was terminated. The initiation of the trial was followed by a random delay, ranging from 250 to 1000 msec, that was included to minimize anticipation of stimulus onset after which the A, V, or AV stimulus was presented. In response, the monkey had to reach a pad just above the starting lever with the left hand. RTs were measured as the latency at which the monkey initiated its motor response (i.e., when it lifted its hand from the lever). We would note that motor kinematics were not recorded or analyzed here. If this RT was initiated within a restricted time window (800 msec), the animal received the reward (pellet). The monkey could initiate the next trial by repositioning the left hand on the start lever and so on. If the monkey did not respond fast enough, the trial was aborted, no reward was delivered, and a delay of 10 sec was imposed before giving the monkey the possibility to move on to the next trial. Trial sequences were randomized across stimulus types within a block. The monkeys were not food deprived, but the task was executed in the morning, thus representing the first access to food on each day. After training, the monkeys received additional food (cereals, fruits). The weight of the animal was monitored daily (MK1 weighed 3.5 kg and MK2 weighed 5 kg). A loss of weight of 10% or more was established as a criterion for interruption; a situation that did not occur in the present study.

The monkeys were initially trained in several steps over a period of 10–14 months to execute the entire auditory–visual–motor task with the head restrained. First, they were trained to press the starting lever and receive a reward. Next, they were trained to touch the target pad when a stimulus was delivered (first, a visual stimulus; second an auditory stimulus; third, an auditory–visual stimulus; and finally, any of the three stimuli presented randomly) and receive a reward. Finally, after implantation of head fixation bars (see below), the monkey was trained to execute the same task with the head fixed and under conditions of controlled eye position.

The duration of both visual and auditory stimuli was fixed at 250 msec throughout the study. The visual stimulus consisted of a 32° square flash delivered on the computer monitor in front of the animal at several intensities defined by 80%, 75%, 60%, 45%, 25%, and 5% levels of contrast, corresponding to 17, 15.2, 9.2, 6.5, 4.6, and 3.6 lx, respectively measured with a Luxmeter ELV LM-300. The auditory stimulus was a white noise burst delivered simultaneously from two loudspeakers placed on each side of the computer monitor at the following intensity levels: 60, 50, 40, 30, 20, 10, 9, 7 dB SPL measured with Sonometer Bruel & Kjaer Type 2231. In the AV conditions, the auditory and the visual stimuli were presented simultaneously. A test (a given combination of intensities) usually consisted of 200 trials, randomly and equally distributed between the A, V, and AV conditions. On a daily session, usually one to three tests were conducted, corresponding to different combinations of intensities. The number of tests considered for each condition was at least 2, and the number of trials considered for each condition was around 200.

The behavioral performance of the monkey (collecting the RTs, checking that the motor response was time locked to the presentation of stimulus, delivery of the reward, etc.) and the controlled presentation of the A, V, or AV stimuli were conducted in the Tucker-Davis-Technology (TDT, Florida, USA) environment on a personal computer using the software “OpenX.” A “correct” response was defined as a monkey's reaching movement aimed to the target (touch the target pad) initiated within a time window of 200 to 800 msec after the presentation of the stimulus (see Kermadi, Liu, Tempini, Calciati, & Rouiller, 1998). Trials on which the RT was faster than 200 msec were considered anticipatory, and trials on which the RT was slower than 800 msec were considered as failure to comply with task demands. Likewise, trials on which the monkey failed to maintain fixation were terminated. In this regard, all these trials were labeled as “incorrect.” The detection threshold was considered as the intensity for which the percentage of correct responses amounted to at least 75%, consistent with discrimination thresholds defined under the rubric of signal detection theory (Macmillan & Creelman, 1991). RTs were analyzed with ANOVA and post hoc tests, as described more precisely in the Results section. When appropriate, Bonferroni correction for multiple comparisons was applied.

Whether the RSE exceeded the facilitation predicted by probability summation was tested using Miller's so-called “race model” inequality (Miller, 1982). Two classes of models have been formulated to account for the RSE: race and coactivation models. According to the race model (Raab, 1962), neural convergence and interactions are not required to obtain the RSE. Rather, stimuli independently compete for response initiation and the faster of the two stimuli mediates the behavioral response on any given trial. Under this model, probability summation can account for the RSE because the likelihood of either of the two stimuli yielding a fast RT on any given trial is higher than that from either stimulus alone. In contrast, according to the coactivation model (e.g., Miller, 1982), neural responses to multisensory stimuli interact before the initiation of the behavioral response. The threshold for initiating a motor response is thus met more rapidly following multisensory than unisensory stimuli. Miller's inequality is a quantitative assessment of whether RTs of a given speed satisfy the condition of probability summation. Violation of this inequality is considered a benchmark for integrative processes, although the converse (i.e., no interactions when probability summation accounts for the data) cannot be claimed.

Formally, this inequality tests whether the probability of an RT of a given speed to a multisensory stimulus is higher than the summed probabilities for an equally fast RT to either unisensory stimulus alone (minus their joint probability under the assumption of complete independence). This entailed calculating the cumulative probability distribution for each condition. To facilitate comparisons across stimulus intensities (as well as across species), these distributions were first divided into 5% bins on the basis of the range of RTs across stimulus conditions within a given test. A model of the probability distribution for each multisensory combination was then calculated. For each 5% bin, the modeled value equals the sum of the probabilities for each component unisensory condition minus their joint probability [i.e., P(RT(A)) + P(RT(V)) − (P(RT(A)) × P(RT(V)))] as in Corballis (1998).

Surgery

The head of the monkey was restrained to allow eye position monitoring. Maintenance of gaze fixation during the trial is important because there is evidence that responses to acoustic stimuli in the auditory midbrain and cortex are influenced by eye position (e.g., Porter, Metzger, & Groh, 2006, 2007; Fu et al., 2004; Werner-Reiss et al., 2003). When the monkeys reached a daily stable level of performance (80–90% of correct responses), two stainless steel head-fixation cylinders were implanted longitudinally above the midline, one at the level of the frontal region and one at the level of the occipital region. The head fixation device was anchored to the skull using 10 titanium screws and orthopedic cement (Palacos R-40 + 500 mg gentamicin).

Surgeries for the implantation of the head-fixation bars were performed as follows under aseptic conditions. The monkeys were preanesthetized with ketamine (5 mg/kg im) and treated with the analgesic Carprofen (Rymadil; 4 mg/kg sc), antibiotics (Albipen: ampicillin 10%, 15–30 mg/kg sc), atropine sulfate (0.05 ml/kg im), and dexamethasone (Decadron, 0.05 ml/kg diluted 1:1 in saline im). Then, they were anaesthetized with a continuous perfusion through the femoral vein (0.1 ml/min/kg) of a mixture of propofol (1%) and 4% glucose solution to which ketamine was added (65 mg/100 ml). During surgery, heart and respiration rates, rectal temperature, O2 blood saturation, and expiratory CO2 content were continuously monitored. After surgery, muscles and skin were sutured, and the animal was treated for several days with an analgesic (pills of Rymadil mixed with food; 5 mg/kg po) and an antibiotic (amoxicillin; 10 mg/kg po). All measures were taken to minimize pain and discomfort for the monkeys.

The monkey was then retrained to perform the task having the head fixed and to fixate a target in the middle of the screen during the task. In a further step, we implanted a recording chamber for single-unit recordings in the auditory cortex (electrophysiological data will be reported elsewhere).

RESULTS

Unisensory Thresholds

In a first phase of the experiment (Figure 2), the auditory and the visual thresholds were assessed by pooling different sessions corresponding to distinct intensity levels. As expected, the higher the intensity of the auditory or the visual stimulus, the higher the probability of a correct response and the shorter the RT (Figure 2). As single-unit recordings will be aimed at the auditory cortex (results reported in a separate publication), particular emphasis was put here on the intensity of the auditory stimulus. Auditory threshold was reached when increasing the intensity from 9 to 10 dB SPL in MK1 and from 10 to 20 dB SPL in MK2 (Figure 2A). As a function of increasing intensities, there was a progressive decrease of RTs in response to the auditory stimulus in both monkeys (Figure 2B). A univariate ANOVA with RTs as a function of the intensity of the auditory stimuli revealed that there was a significant effect of the intensity of the auditory stimuli on RTs (p < .0001 for both monkeys). Post hoc unpaired t tests confirmed these RT differences for distinct auditory intensities, except for the RTs obtained in response to 50 and 60 dB SPL in MK2 (p = .61). In addition, it is noteworthy that the RTs were substantially shorter and less variable in MK1 than in MK2, irrespective of the noise burst's intensity, because it is MK2 who exhibits the larger gain from multisensory stimuli (see Figure 5 and The impact of stimulus intensity on multisensory gains and costs section).

Figure 2. 

Evaluation of auditory (panel A) and visual (panel C) thresholds for monkey 1 (MK1, filled square) and monkey 2 (MK2, open circle), as derived from multiple sessions: In each individual session, one intensity for each mode, visual (V) and auditory (A), was tested. Panels A and C represent the percentage of correct responses generated by each monkey as a function of stimulus intensity in the A and V conditions, respectively. The detection threshold is defined as the intensity at which a percentage of correct responses reached 75% (horizontal dashed line). In panels B and D, the mean RT and its SEM was plotted as a function of stimulus intensity for MK1 and MK2, in the A and V conditions, respectively. For each data point, the number of trials (A, C) or RTs (B, D) ranged between 80 and 430.

Figure 2. 

Evaluation of auditory (panel A) and visual (panel C) thresholds for monkey 1 (MK1, filled square) and monkey 2 (MK2, open circle), as derived from multiple sessions: In each individual session, one intensity for each mode, visual (V) and auditory (A), was tested. Panels A and C represent the percentage of correct responses generated by each monkey as a function of stimulus intensity in the A and V conditions, respectively. The detection threshold is defined as the intensity at which a percentage of correct responses reached 75% (horizontal dashed line). In panels B and D, the mean RT and its SEM was plotted as a function of stimulus intensity for MK1 and MK2, in the A and V conditions, respectively. For each data point, the number of trials (A, C) or RTs (B, D) ranged between 80 and 430.

Visual threshold was less accurately determined. It was at slightly above 5% of contrast in MK1, whereas threshold was already exceeded at the lowest intensity tested in MK2 (Figure 2C). RTs to the visual stimulus were less affected by the intensity levels we used (Figure 2D). The univariate ANOVA on RTs as a function of the visual intensity showed that there was significant effect of the intensity of the visual stimuli on RTs for MK2 (p < .0001) but no significant effect for MK1. Post hoc unpaired t tests revealed that this effect was only due to a significant difference between RTs for 5% contrast versus all the other contrast levels (p < .0001), the latter of which did not differ among each other.

In summary, these results show that the range of auditory stimulus intensities tested here consistently modulated the rate of correct responses and/or RTs. Such was not the case for the visual stimulus intensities we examined. The subsequent portions of the experiment therefore systematically varied the intensity of the auditory stimulus, whereas that of the visual stimulus was always at least 25% contrast (i.e., at levels where performance was no longer below the 75% accuracy threshold; see Figure 2).

The Impact of Stimulus Intensity on the RSE

Accuracy rates for both monkeys across stimulus conditions and as a function of stimulus intensity are listed in Table 1. Mean RTs in response to A, V, and AV stimuli are shown for each monkey across a range of stimulus parameters (Figure 3 for MK1 and MK2). Each pairing of a given auditory and visual intensity was analyzed with a univariate repeated measures ANOVA with the within-subject factor of stimulus condition (A, V, and AV). The results of these analyses are summarized in Tables 2 and 3 for MK1 and MK2, respectively. Pairings involving noise bursts at near threshold and up to 20 dB above threshold levels resulted in a significant RSE in both monkeys, whereas pairings involving noise bursts 30 dB or more above threshold levels failed to produce a significant RSE (i.e., RTs were no different between AV and A conditions). This pattern of results was observed irrespective of the contrast level of the visual stimulus. It is also worth noting that RTs to auditory stimuli were significantly slower than those to visual stimuli when the sound intensity was just above threshold (10 dB in MK1 and 20 dB in MK2), again irrespective of the visual contrast level, but that this pattern reversed (i.e., auditory RTs became faster than visual RTs) when the sound intensity was increased. In other words, at suprathreshold intensity and contrast levels, RTs to auditory stimuli were significantly faster than those to visual stimuli, a finding consistent with observations in humans (e.g., Hershenson, 1962; Todd, 1912).

Table 1. 

Accuracy Rates

Stimuli
MK1
MK2
AV
A
V
AV
A
V
Threshold 88 62 87 90 65 80 
10 dB above threshold 91 90 89 92 85 88 
20 dB above threshold 87 86 88 86 89 86 
30 dB above threshold 86 80 83 93 76 87 
40 dB above threshold 88 82 89 82 78 90 
Stimuli
MK1
MK2
AV
A
V
AV
A
V
Threshold 88 62 87 90 65 80 
10 dB above threshold 91 90 89 92 85 88 
20 dB above threshold 87 86 88 86 89 86 
30 dB above threshold 86 80 83 93 76 87 
40 dB above threshold 88 82 89 82 78 90 
Figure 3. 

RT data for MK1 (left column) and for MK2 (right column) at moderate auditory intensities (at threshold, 10 and 20 dB above) and at high auditory intensities (30 and 40 dB above threshold). Mean RTs and SEM are indicated for auditory, visual, and auditory–visual stimuli for different intensity conditions. Green bars correspond to visual (V), blue to auditory (A), and red to auditory–visual (AV) stimuli. Stimulus parameters are indicated on top of each graph.

Figure 3. 

RT data for MK1 (left column) and for MK2 (right column) at moderate auditory intensities (at threshold, 10 and 20 dB above) and at high auditory intensities (30 and 40 dB above threshold). Mean RTs and SEM are indicated for auditory, visual, and auditory–visual stimuli for different intensity conditions. Green bars correspond to visual (V), blue to auditory (A), and red to auditory–visual (AV) stimuli. Stimulus parameters are indicated on top of each graph.

Table 2. 

Analyses of RTs from MK1

Stimuli
Main Effect of Stimulus Condition
Post hoc Contrasts
AV vs. A
AV vs. V
A vs. V
Threshold F(2, 178) = 70.164, p < .001 t(348) = 6.16, p < .001 t(378) = 4.14, p < .001 t(328) = 3.52, p < .001 
10 dB above threshold F(2, 244) = 168.627, p < .001 t(363) = 4.77, p < .001 t(354) = 12.91, p < .001 t(347) = 3.53, p < .001 
20 dB above threshold F(2, 303) = 132.976, p < .001 t(658) = 5.95, p < .001 t(753) = 25.07, p < .001 t(703) = 14.07, p < .001 
30 dB above threshold F(2, 163) = 155.113, p < .001 t(288) = 1.78, p = .077 t(288) = 8.99, p < .001 t(290) = 9.38, p < .001 
40 dB above threshold F(2, 156) = 117.573, p < .001 t(284) = 0.56, p = .575 t(253) = 10.95, p < .001 t(243) = 7.86, p < .001 
Stimuli
Main Effect of Stimulus Condition
Post hoc Contrasts
AV vs. A
AV vs. V
A vs. V
Threshold F(2, 178) = 70.164, p < .001 t(348) = 6.16, p < .001 t(378) = 4.14, p < .001 t(328) = 3.52, p < .001 
10 dB above threshold F(2, 244) = 168.627, p < .001 t(363) = 4.77, p < .001 t(354) = 12.91, p < .001 t(347) = 3.53, p < .001 
20 dB above threshold F(2, 303) = 132.976, p < .001 t(658) = 5.95, p < .001 t(753) = 25.07, p < .001 t(703) = 14.07, p < .001 
30 dB above threshold F(2, 163) = 155.113, p < .001 t(288) = 1.78, p = .077 t(288) = 8.99, p < .001 t(290) = 9.38, p < .001 
40 dB above threshold F(2, 156) = 117.573, p < .001 t(284) = 0.56, p = .575 t(253) = 10.95, p < .001 t(243) = 7.86, p < .001 
Table 3. 

Analyses of RTs from MK2

Stimuli
Main Effect of Stimulus Condition
Post hoc Contrasts
AV vs. A
AV vs. V
A vs. V
Threshold F(2, 52) = 8.422, p < .001 t(112) = 4.79, p < .001 t(133) = 0.07, p < .001 t(127) = 5.14, p < .001 
10 dB above threshold F(2, 166) = 41.367, p < .001 t(344) = 5.57, p < .001 t(334) = 8.24, p < .001 t(344) = 2.26, p < .001 
20 dB above threshold F(2, 68) = 53.128, p < .001 t(144) = 2.59, p < .001 t(139) = 7.24, p < .001 t(145) = 4.20, p < .001 
30 dB above threshold F(2, 113) = 89.736, p < .001 t(247) = 0.26, p = .798 t(251) = 16.68, p < .001 t(232) = 14.92, p < .001 
40 dB above threshold F(2, 63) = 27.43, p < .001 t(153) = 0.72, p = .473 t(138) = 4.31, p < .001 t(143) = 5.39, p < .001 
Stimuli
Main Effect of Stimulus Condition
Post hoc Contrasts
AV vs. A
AV vs. V
A vs. V
Threshold F(2, 52) = 8.422, p < .001 t(112) = 4.79, p < .001 t(133) = 0.07, p < .001 t(127) = 5.14, p < .001 
10 dB above threshold F(2, 166) = 41.367, p < .001 t(344) = 5.57, p < .001 t(334) = 8.24, p < .001 t(344) = 2.26, p < .001 
20 dB above threshold F(2, 68) = 53.128, p < .001 t(144) = 2.59, p < .001 t(139) = 7.24, p < .001 t(145) = 4.20, p < .001 
30 dB above threshold F(2, 113) = 89.736, p < .001 t(247) = 0.26, p = .798 t(251) = 16.68, p < .001 t(232) = 14.92, p < .001 
40 dB above threshold F(2, 63) = 27.43, p < .001 t(153) = 0.72, p = .473 t(138) = 4.31, p < .001 t(143) = 5.39, p < .001 

The Impact of Stimulus Intensity on Psychophysical Metrics of Multisensory Interactions

Miller's race model inequality (Miller, 1982) was used to assess whether neural response interactions need be invoked to explain the above RSEs. As described in the Methods section, for each pairing of stimulus intensities, the cumulative probability functions was calculated for each condition (A, V, and AV). The values from the unisensory conditions were in turn used to establish boundary values on the basis of probability summation, beyond which (i.e., to the left of which) neural response interactions need be invoked. These cumulative probability functions are displayed in Figure 4A and B, respectively, for one representative condition for each monkey. In these plots, faster RTs are evident as leftward shifts (i.e., higher probabilities at a given RT latency). Pairings involving noise bursts at near threshold and up to 20 dB above threshold levels resulted in a leftward shift of the cumulative probability function for AV stimuli (i.e., red curves in Figure 4A and B for an example), whereas pairings involving noise bursts 30 dB or more above threshold levels failed to do so. In fact, at these auditory intensities, the cumulative probability function was often more leftward for A than for AV stimuli. As was the case with mean RTs, this pattern of results was observed irrespective of the contrast level of the visual stimulus. It is also worth noting that the leftward shift of the cumulative probability function for AV stimuli was more pronounced in MK2 than in MK1, indicating that multisensory facilitation was more pronounced in the monkey characterized by generally slower RTs to the unisensory stimuli.

Figure 4. 

Exemplar cumulative probability distributions of the RTs in the three conditions (A, V, and AV) at a moderate auditory intensity for each monkey (panel A for MK1 and panel B for MK2). The dashed curve displays values predicted according by Miller's (1982) race model inequality. (C, D) Graphical representations after applying Miller's inequality (see Methods) to each set of stimulus parameters (indicated in the legend). This representation indicates whether the race model is violated (positive values) or satisfied (negative values) for each monkey (panel C for MK1 and panel D for MK2).

Figure 4. 

Exemplar cumulative probability distributions of the RTs in the three conditions (A, V, and AV) at a moderate auditory intensity for each monkey (panel A for MK1 and panel B for MK2). The dashed curve displays values predicted according by Miller's (1982) race model inequality. (C, D) Graphical representations after applying Miller's inequality (see Methods) to each set of stimulus parameters (indicated in the legend). This representation indicates whether the race model is violated (positive values) or satisfied (negative values) for each monkey (panel C for MK1 and panel D for MK2).

Comparison of the cumulative probability functions with AV stimuli and the modeled values was performed for each pairing of stimulus intensities to determine whether and over which portions of the RT distribution neural response interactions need be invoked (Figure 4C and D, respectively, for each monkey). Such violation of the race model inequality would be indicative of RT facilitation in excess of predictions on the basis of probability summation. For the conditions near threshold and just above (at 10, 20, and 30 dB SPL for MK1 and at 20, 30, and 40 dB SPL for MK2), we observed a violation of the race model (i.e., positive values) over the faster half of the RT distribution. By contrast and in agreement with the above analyses of mean RTs, the race model was respected at higher sound intensities (zero or negative values), providing no necessity to invoke neural response interactions (although nonlinear neural response interactions in the absence of an RSE exceeding probability summation have been previously observed in Murray, Foxe, Higgins, Javitt, & Schroeder, 2001).

The Impact of Stimulus Intensity on Multisensory Gains and Costs

Figure 5 illustrates the multisensory gain obtained in the AV condition as compared with the A or V conditions alone, as a function of auditory intensity. The multisensory gain/cost was defined as the decrease (in percent) of the mean RT obtained in the multisensory condition as compared with the mean RT obtained in the best unisensory condition (Stein & Meredith, 1993). In MK1 (squares in Figure 5), gains (i.e., facilitative RSEs) were observed at auditory intensities from threshold to 20 dB above threshold and ranged from 5% to 7%. In MK2 (open circles in Figure 5), gains ranged from 7% to 11% and were also observed at auditory intensities from threshold to 20 dB above threshold. By contrast, either no gain or costs were observed at auditory intensities 30–40 dB above threshold in both monkeys. These gains and costs represent the mean across the full distribution of RTs relative to the better unisensory condition. An alternative approach for assessing the gains/costs of multisensory stimuli is provided in the contrast of the observed and modeled cumulative probability distributions at each 5% bin of the RT distribution (Figure 4). These plots show that gains were approximately 4–8% in MK1 and 15–25% in MK2. Whether there is a direct link between the size of the gain obtained and the general efficacy with which the subject performed the task is a speculative possibility that will require additional data from a larger cohort of participants. Nonetheless, the pattern in the present data highlights how inspection of single-subject results in studies of multisensory processes in humans might provide insights on the determinants of performance benefits, particularly when imaging data are simultaneously acquired (e.g., Sperdin et al., 2009).

Figure 5. 

RSE magnitude expressed as a percent of gain or cost following auditory–visual (AV) stimulation as compared with the faster of the unisensory conditions. Filled squares display these values for MK1 as a function of the stimulus conditions shown in Figure 3 (i.e., condition 1 = 10 dB SPL, 25% visual contrast; condition 2 = 20 dB SPL, 60% visual contrast; condition 3 = 30 dB SPL, 75% visual contrast; condition 4 = 40 dB SPL, 45% visual contrast; condition 5 = 50 dB SPL, 80% of contrast). Open circles display the values for MK2 as a function of the stimulus conditions shown in Figure 3 (i.e., Condition 1 = 20 dB SPL, 60% visual contrast; Condition 2 = 30 dB SPL, 45% visual contrast; Condition 3 = 40 dB SPL, 75% visual contrast; Condition 4 = 50 dB SPL, 60% visual contrast; Condition 5 = 60 dB SPL, 60% visual contrast).

Figure 5. 

RSE magnitude expressed as a percent of gain or cost following auditory–visual (AV) stimulation as compared with the faster of the unisensory conditions. Filled squares display these values for MK1 as a function of the stimulus conditions shown in Figure 3 (i.e., condition 1 = 10 dB SPL, 25% visual contrast; condition 2 = 20 dB SPL, 60% visual contrast; condition 3 = 30 dB SPL, 75% visual contrast; condition 4 = 40 dB SPL, 45% visual contrast; condition 5 = 50 dB SPL, 80% of contrast). Open circles display the values for MK2 as a function of the stimulus conditions shown in Figure 3 (i.e., Condition 1 = 20 dB SPL, 60% visual contrast; Condition 2 = 30 dB SPL, 45% visual contrast; Condition 3 = 40 dB SPL, 75% visual contrast; Condition 4 = 50 dB SPL, 60% visual contrast; Condition 5 = 60 dB SPL, 60% visual contrast).

DISCUSSION

The present study demonstrates the feasibility and suitability of using simple manual detection tasks in behaving monkeys to investigate links between psychophysical and neurophysiologic instantiations of multisensory interactions. Several aspects of the RSE we obtained in macaque monkeys are qualitatively and quantitatively similar to the RSE observed in humans. Specifically, we show that RTs are facilitated under multisensory conditions in excess of predictions on the basis of probability summation and therefore necessitate the invocation of neural response interactions. Moreover, and in contrast to prior research (Miller et al., 2001; Lamarre et al., 1983), this facilitation was obtained following synchronous presentation of the auditory–visual stimuli. Our parametric variation of auditory stimulus intensity revealed that in both animals, the RSE was limited to situations where the auditory stimulus intensity was below or up to 20 dB above perceptual threshold despite the visual stimulus always being suprathreshold. No RSE or even behavioral costs were obtained with auditory intensities 30–40 dB above threshold. These findings, along with interindividual differences (i.e., the slower of the two monkeys exhibited the larger gain from multisensory stimuli; see Figure 5), are consistent with the principle of inverse effectiveness (PoIE; Stein & Meredith, 1993). In what follows, we situate these findings within the extant psychophysical literature in humans and monkeys as well as current models of multisensory interactions. We conclude by discussing how these results promote new avenues of research that can yield a fuller understanding of the mechanisms of multisensory interactions and integration at scales ranging from single neurons to behavior.

In all cases when a RSE was observed, the facilitation of RTs exceeded predictions on the basis of probability summation. That is, application of Miller's race-model inequality (Miller, 1982) indicated that neural response interactions before the generation of the motor output response need to be occurring to account for the monkeys' performance on multisensory trials. The anatomical level at which behaviorally relevant multisensory convergence occurs is still debated (Miller et al., 2001). Some suggest it to occur during early sensory processing stages level (Gondan, Niederhaus, Rösler, & Röder, 2005; Miller et al., 2001; Mordkoff, Miller, & Roch, 1996), whereas others support the role of motor stages (Giray & Ulrich, 1993). Human brain imaging studies of the relationship between behavioral and neuronal indices (Jepma, Wagenmakers, Band, & Nieuwenhuis, 2009; Romei et al., 2007, 2009; Sperdin et al., 2009; Senkowski, Molholm, Gomez-Ramirez, & Foxe, 2006) suggest that behaviorally relevant multisensory convergence occurs early in sensory processing, before the decision and motor levels (for results in monkeys, see also Wang et al., 2008; Miller et al., 2001; Lamarre et al., 1983). For example, Romei et al. (2007) found that the magnitude of the RSE matched and correlated with the facilitation of RTs induced by the combination of single-pulse TMS over the occipital pole with the presentation of sounds. These data suggest that multisensory interactions within occipital regions may be linked to behavioral indices of RT facilitation. The present findings thus open the possibility of conducting similar research in a nonhuman primate to reveal the mechanisms of such interactions and their behavioral relevance at a finer scale.

Our behavioral results in the monkey are in general accordance with the PoIE of multisensory integration, which describes how the largest percentile gain in neuronal discharge rates is observed when unisensory stimuli are minimally effective and diminishes (or even becomes a cost) when unisensory stimuli are maximally effective (Stein & Meredith, 1993). Our study shows mainly the influence of the auditory stimulus on this principle. The lack of dependence of the effectiveness of the visual stimulus could be explained by the fact that even in the original studies (e.g., Meredith & Stein, 1986), there are facilitative effects with highly effective stimuli. Also, there are data showing that one can see modulations in multisensory gain even if just one of the stimuli is less effective (e.g., Kayser, Petkov, Augath, & Logothetis, 2005; Perrault, Vaughan, Stein, & Wallace, 2005; for a discussion of statistical issues in evaluating the PoIE, see also Holmes, 2009). At a neuronal level (at least within the superior colliculus), supra-additive effects were more likely when the neuron had a small or minimal dynamic range of responsiveness to unisensory stimuli (Perrault et al., 2005). By contrast, additive and subadditive effects were more likely when the neuron had a large dynamic range. In the case of our data, it would seem that both animals exhibited dynamic ranges in performance only in the auditory modality and therefore were in this regard similar to the class of “single modality dynamic range” neurons described in Perrault et al. (2005). Interestingly, this profile was observed in 49% of the tested neurons. In the case of psychophysics, this principle is often applied to the observation of larger benefits of multisensory stimulation when the unisensory stimuli are themselves near threshold and/or noisy (Ma, Zhou, Ross, Foxe, & Parra, 2009; Bolognini, Leo, Passamonti, Stein, & Làdavas, 2007; Ross, Saint-Amour, Leavitt, Javitt, et al., 2007; Rach & Diederich, 2006; Diederich & Colonius, 2004; Grant & Seitz, 2000; Sumby & Pollack, 1954). This principle can also be applied when considering the perceptual benefits of multisensory interactions in the case of sensory deficits (Rouger et al., 2007; Laurienti et al., 2006; Hairston, Laurienti, Mishra, Burdette, & Wallace, 2003). In the case of simple manual RTs to AV stimuli in humans, RSEs (often exceeding probability summation) are nonetheless reliably obtained with suprathreshold stimuli in both sensory modalities (e.g., Martuzzi et al., 2007; Romei et al., 2007; Molholm et al., 2002; Giard & Peronnet, 1999). It will therefore be important for future research to resolve the basis of the apparent discrepancy between effects in monkeys where the RSE was limited to near-threshold conditions (i.e., the present data as well as those of Miller et al., 2001; Lamarre et al., 1983) and effects in humans observed at suprathreshold intensities. A similarly intriguing aspect of the present results that warrants continued investigation is that the RSE was observed irrespective of the suprathreshold contrast level of the visual stimuli. More generally and despite the abovementioned points for future research, the magnitude of the RSE we obtained, which was on the order of a 5–10% gain in RTs, is highly consistent with that typically observed in the abovementioned studies in humans albeit under different stimulation paradigms. It will therefore be particularly informative for all of the above issues to conduct identical psychophysical paradigms (and if possible brain imaging or brain mapping paradigms) in both humans and monkeys. The present results provide an important first step in this direction.

Additional data provide insights on potential neural substrates to account for RSEs necessitating neural response interactions at a sensory rather than a motor level. Anatomical data in monkeys have documented the existence of direct projections between areas otherwise considered unisensory (for a review, see Cappe, Rouiller, & Barone, 2009; Cappe & Barone, 2005; Rockland & Ojima, 2003; Falchier, Clavagnier, Barone, & Kennedy, 2002), which could support multisensory interactions at low hierarchical stages. At a cortical level, multisensory convergence is present in primary sensory cortices. On the one hand, the core auditory cortex in monkeys receives direct visual projections (Cappe & Barone, 2005). On the other hand, visual areas such as V1 or V2 receive direct inputs from the core auditory cortex (Smiley & Falchier, 2009; Falchier et al., 2002). Moreover, recent findings also implicate the thalamus in multisensory interactions (for a review, see Cappe, Morel, Barone, & Rouiller, 2009; Cappe, Rouiller, et al., 2009; Cappe, Morel, & Rouiller, 2007; Hackett et al., 2007; Komura, Tamura, Uwano, Nishijo, & Ono, 2005). Although the requisite studies directly linking such anatomical data to behavior are still lacking, it is noteworthy that some nuclei in the thalamus, such as the medial pulvinar, receive either mixed sensory inputs or projections from different sensory cortical areas and in turn project to sensory and premotor areas (for a review, see Cappe, Morel, et al., 2009; Cappe, Rouiller, et al., 2009). Sensory modalities may already interact at a thalamic level before being directly conveyed to the premotor cortex and consequently contributing to facilitated behavior.

At a neurophysiologic level, numerous studies in animals have described a large set of cortical areas of the frontal, temporal, or parietal lobes in which the neuronal activity is related to multisensory processing (e.g., Chandrasekaran & Ghazanfar, 2009; Sugihara, Diltz, Averbeck, & Romanski, 2006; Barraclough, Xiao, Baker, Oram, & Perrett, 2005; for reviews, see Stein & Stanford, 2008; Romanski, 2007; Ghazanfar & Schroeder, 2006; Schroeder & Foxe, 2002; Stein & Meredith, 1993). Until recently, the predominant model was that multisensory interactions were restricted to higher order association cortices and the superior colliculus (although multisensory responses in this latter structure have been shown to depend on cortical inputs; Alvarado, Stanford, Vaughan, & Stein, 2007). Nowadays, there is increasing evidence in both animals and humans that multisensory interactions capitalize upon the abovementioned anatomical connectivity, such that effects are observed within primary and similarly low levels at early poststimulus latencies (e.g., Meredith & Allman, 2009; Ghazanfar, Chandrasekaran, & Logothetis, 2008; Kayser et al., 2008; Wang et al., 2008; Bizley, Nodal, Bajo, Nelken, & King, 2007; Kayser, Petkov, Augath, & Logothetis, 2007; Martuzzi et al., 2007; Brosch, Selezneva, & Scheich, 2005; Kayser et al., 2005; Fu et al., 2003; Molholm et al., 2002; Schroeder et al., 2001; Giard & Peronnet, 1999; reviewed in Driver & Noesselt, 2008; Ghazanfar & Schroeder, 2006). It is worth noting that auditory inputs into primary visual cortex and conversely visual inputs into core auditory cortex are believed to serve a predominantly modulatory function, acting on the oscillatory activity (Kayser et al., 2008; Schroeder, Lakatos, Kajikawa, Partan, & Puce, 2008; Lakatos, Chen, O'Connell, Mills, & Schroeder, 2007) or on the latency of the neuronal responses (Wang et al., 2008). Both such mechanisms could serve to enhance the speed of sensory processing and induce a reduction of RTs following multisensory stimulation. In the latter study (Wang et al., 2008), neurons recorded in the primary visual cortex exhibited a significant reduction in visual response latencies specifically under suboptimal conditions; a result that mimics the behavioral data of the present study. Although it is unlikely that the benefit in detection evinced in the present study can be ascribed solely to such changes in brain activity, the universal nature of the manner in which multisensory neurons throughout the brain process their sensory cues makes this a reasonable mechanistic model.

Conclusion

Given the dramatic growth in multisensory research over the past decades as well as in the recognition of the role of multisensory processes in human development, aging, and disease (for reviews, see Driver & Noesselt, 2008; Stein & Stanford, 2008), there is a clear and present need for an animal model that bears high resemblance not only neuroanatomically and neurophysiologically (as has already been established over decades of research) but also behaviorally (for studies in rats, see Hirokawa, Bosch, Sakata, Sakurai, & Yamamori, 2008; Komura et al., 2005). The present study demonstrates that macaque monkeys do exhibit multisensory performance gains that are highly similar to those observed in humans when completing a stimulus detection task via arm movements. Both the paradigm and the behavioral outcome provide an essential first step for future studies of the neurophysiologic bases of multisensory processes.

Acknowledgments

The authors thank technical assistance of Dr. Gérard Loquet (software programming), Josef Corpataux, Bernard Bapst, Laurent Bossy and Bernard Morandi (animal house keeping), André Gaillard (mechanics), Bernard Aebischer (electronics), and Laurent Monney (informatics).

Grant sponsors: The Swiss National Science Foundation, grant nos. 31-61857.00 (E. M. R.), 310000-110005 (E. M. R.), and 3100AO-118419 (M. M. M.); the Novartis Foundation; and the Swiss National Science Foundation Centre of Competence in Research (NCCR) on “Neural plasticity and repair.”

Reprint requests should be sent to Céline Cappe, The Functional Electrical Neuroimaging Laboratory, Neuropsychology and Neurorehabilitation Service and Radiology Service, Centre Hospitalier Universitaire Vaudois and University of Lausanne, Rue du Bugnon 46, CH-1011 Lausanne, Switzerland, or via e-mail: Celine.Cappe@chuv.ch.

REFERENCES

Alvarado
,
J. C.
,
Stanford
,
T. R.
,
Vaughan
,
J. W.
, &
Stein
,
B. E.
(
2007
).
Cortex mediates multisensory but not unisensory integration in superior colliculus.
Journal of Neuroscience
,
27
,
12775
12786
.
Barraclough
,
N. E.
,
Xiao
,
D.
,
Baker
,
C. I.
,
Oram
,
M. W.
, &
Perrett
,
D. I.
(
2005
).
Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions.
Journal of Cognitive Neuroscience
,
17
,
377
391
.
Bell
,
A. H.
,
Corneil
,
B. D.
,
Meredith
,
M. A.
, &
Munoz
,
D. P.
(
2001
).
The influence of stimulus properties on multisensory processing in the awake primate superior colliculus.
Canadian Journal of Experimental Psychology
,
55
,
123
132
.
Bell
,
A. H.
,
Meredith
,
M. A.
,
Van Opstal
,
A. J.
, &
Munoz
,
D. P.
(
2005
).
Crossmodal integration in the primate superior colliculus underlying the preparation and initiation of saccadic eye movements.
Journal of Neurophysiology
,
93
,
3659
3673
.
Bizley
,
J. K.
,
Nodal
,
F. R.
,
Bajo
,
V. M.
,
Nelken
,
I.
, &
King
,
A. J.
(
2007
).
Physiological and anatomical evidence for multisensory interactions in auditory cortex.
Cerebral Cortex
,
17
,
2172
2189
.
Blau
,
V.
,
van Atteveldt
,
N.
,
Ekkebus
,
M.
,
Goebel
,
R.
, &
Blomert
,
L.
(
2009
).
Reduced neural integration of letters and speech sounds links phonological and reading deficits in adult dyslexia.
Current Biology
,
19
,
503
508
.
Bolognini
,
N.
,
Leo
,
F.
,
Passamonti
,
C.
,
Stein
,
B. E.
, &
Làdavas
,
E.
(
2007
).
Multisensory-mediated auditory localization.
Perception
,
36
,
1477
1485
.
Brosch
,
M.
,
Selezneva
,
E.
, &
Scheich
,
H.
(
2005
).
Nonauditory events of a behavioral procedure activate auditory cortex of highly trained monkeys.
Journal of Neuroscience
,
25
,
6797
6806
.
Cappe
,
C.
, &
Barone
,
P.
(
2005
).
Heteromodal connections supporting multisensory integration at low levels of cortical processing in the monkey.
European Journal of Neuroscience
,
22
,
2886
2902
.
Cappe
,
C.
,
Morel
,
A.
,
Barone
,
P.
, &
Rouiller
,
E. M.
(
2009
).
The thalamocortical projection systems in primate: An anatomical support for multisensory and sensorimotor integrations.
Cerebral Cortex
,
19
,
2025
2037
.
Cappe
,
C.
,
Morel
,
A.
, &
Rouiller
,
E. M.
(
2007
).
Thalamocortical and the dual pattern of corticothalamic projections of the posterior parietal cortex in macaque monkeys.
Neuroscience
,
146
,
1371
1387
.
Cappe
,
C.
,
Rouiller
,
E. M.
, &
Barone
,
P.
(
2009
).
Multisensory anatomic pathway.
Hearing Research
,
258
,
28
36
.
Cappe
,
C.
,
Thut
,
G.
,
Romei
,
V.
, &
Murray
,
M. M.
(
2009
).
Selective integration of auditory–visual looming cues by humans.
Neuropsychologia
,
47
,
1045
1052
.
Chandrasekaran
,
C.
, &
Ghazanfar
,
A. A.
(
2009
).
Different neural frequency bands integrate faces and voices differently in the superior temporal sulcus.
Journal of Neurophysiology
,
101
,
773
788
.
Corballis
,
M. C.
(
1998
).
Interhemispheric neural summation in the absence of the corpus callosum.
Brain
,
121
,
1795
1807
.
Corneil
,
B. D.
,
Van Wanrooij
,
M.
,
Munoz
,
D. P.
, &
Van Opstal
,
A. J.
(
2002
).
Auditory-visual interactions subserving goal-directed saccades in a complex scene.
Journal of Neurophysiology
,
88
,
438
454
.
Diederich
,
A.
, &
Colonius
,
H.
(
2004
).
Bimodal and trimodal multisensory enhancement: Effects of stimulus onset and intensity on RT.
Perception and Psychophysics
,
66
,
1388
1404
.
Driver
,
J.
, &
Noesselt
,
T.
(
2008
).
Multisensory interplay reveals crossmodal influences on “sensory-specific” brain regions, neural responses, and judgments.
Neuron
,
57
,
11
23
.
Durif
,
C.
,
Jouffrais
,
C.
, &
Rouiller
,
E. M.
(
2003
).
Single-unit responses in the auditory cortex of monkeys performing a conditional acousticomotor task.
Experimental Brain Research
,
153
,
614
627
.
Falchier
,
A.
,
Clavagnier
,
S.
,
Barone
,
P.
, &
Kennedy
,
H.
(
2002
).
Anatomical evidence of multimodal integration in primate striate cortex.
Journal of Neuroscience
,
22
,
5749
5759
.
Frens
,
M. A.
, &
Van Opstal
,
A. J.
(
1998
).
Visual–auditory interactions modulate saccade-related activity in monkey superior colliculus.
Brain Research Bulletin
,
46
,
211
224
.
Fu
,
K. M.
,
Johnston
,
T. A.
,
Shah
,
A. S.
,
Arnold
,
L.
,
Smiley
,
J.
,
Hackett
,
T. A.
,
et al
(
2003
).
Auditory cortical neurons respond to somatosensory stimulation.
Journal of Neuroscience
,
23
,
7510
7515
.
Fu
,
K. M.
,
Shah
,
A. S.
,
O'Connell
,
M. N.
,
McGinnis
,
T.
,
Eckholdt
,
H.
,
Lakatos
,
P.
,
et al
(
2004
).
Timing and laminar profile of eye-position effects on auditory responses in primate auditory cortex.
Journal of Neurophysiology
,
92
,
3522
3531
.
Ghazanfar
,
A. A.
,
Chandrasekaran
,
C.
, &
Logothetis
,
N. K.
(
2008
).
Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys.
Journal of Neuroscience
,
28
,
4457
4469
.
Ghazanfar
,
A. A.
, &
Logothetis
,
N. K.
(
2003
).
Facial expressions linked to monkey calls.
Nature
,
423
,
937
938
.
Ghazanfar
,
A. A.
,
Maier
,
J. X.
,
Hoffman
,
K. L.
, &
Logothetis
,
N. K.
(
2005
).
Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex.
Journal of Neuroscience
,
25
,
5004
5012
.
Ghazanfar
,
A. A.
,
Nielsen
,
K.
, &
Logothetis
,
N. K.
(
2006
).
Eye movements of monkeys viewing vocalizing conspecifics.
Cognition
,
101
,
515
529
.
Ghazanfar
,
A. A.
, &
Schroeder
,
C. E.
(
2006
).
Is neocortex essentially multisensory?
Trends in Cognitive Sciences
,
10
,
278
285
.
Giard
,
M. H.
, &
Peronnet
,
F.
(
1999
).
Auditory–visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study.
Journal of Cognitive Neuroscience
,
11
,
473
490
.
Giray
,
M.
, &
Ulrich
,
R.
(
1993
).
Motor coactivation revealed by response force in divided and focused attention.
Journal of Experimental Psychology: Human Perception and Performance
,
19
,
1278
1291
.
Gondan
,
M.
,
Niederhaus
,
B.
,
Rösler
,
F.
, &
Röder
,
B.
(
2005
).
Multisensory processing in the redundant-target effect: A behavioral and event-related potential study.
Perception and Psychophysics
,
67
,
713
726
.
Grant
,
K. W.
, &
Seitz
,
P. F.
(
2000
).
The use of visible speech cues for improving auditory detection of spoken sentences.
Journal of the Acoustical Society of America
,
108
,
1197
1208
.
Hackett
,
T. A.
,
De La Mothe
,
L. A.
,
Ulbert
,
I.
,
Karmos
,
G.
,
Smiley
,
J.
, &
Schroeder
,
C. E.
(
2007
).
Multisensory convergence in auditory cortex, II. Thalamocortical connections of the caudal superior temporal plane.
Journal of Comparative Neurology
,
502
,
924
952
.
Hairston
,
W. D.
,
Burdette
,
J. H.
,
Flowers
,
D. L.
,
Wood
,
F. B.
, &
Wallace
,
M. T.
(
2005
).
Altered temporal profile of visual–auditory multisensory interactions in dyslexia.
Experimental Brain Research
,
166
,
474
480
.
Hairston
,
W. D.
,
Laurienti
,
P. J.
,
Mishra
,
G.
,
Burdette
,
J. H.
, &
Wallace
,
M. T.
(
2003
).
Multisensory enhancement of localization under conditions of induced myopia.
Experimental Brain Research
,
152
,
404
408
.
Hershenson
,
M.
(
1962
).
Reaction time as a measure of intersensory facilitation.
Journal of Experimental Psychology
,
63
,
289
293
.
Hirokawa
,
J.
,
Bosch
,
M.
,
Sakata
,
S.
,
Sakurai
,
Y.
, &
Yamamori
,
T.
(
2008
).
Functional role of the secondary visual cortex in multisensory facilitation in rats.
Neuroscience
,
153
,
1402
1417
.
Holmes
,
N. P.
(
2009
).
The principle of inverse effectiveness in multisensory integration: Some statistical considerations.
Brain Topography
,
21
,
168
176
.
Hughes
,
H. C.
,
Reuter-Lorenz
,
P. A.
,
Nozawa
,
G.
, &
Fendrich
,
R.
(
1994
).
Visual–auditory interactions in sensorimotor processing: Saccades versus manual responses.
Journal of Experimental Psychology: Human Perception and Performance
,
20
,
131
153
.
Izumi
,
A.
, &
Kojima
,
S.
(
2004
).
Matching vocalizations to vocalizing faces in a chimpanzee (Pan troglodytes).
Animal Cognition
,
7
,
179
184
.
Jepma
,
M.
,
Wagenmakers
,
E. J.
,
Band
,
G. P.
, &
Nieuwenhuis
,
S.
(
2009
).
The effects of accessory stimuli on information processing: Evidence from electrophysiology and a diffusion model analysis.
Journal of Cognitive Neuroscience
,
21
,
847
864
.
Kayser
,
C.
, &
Logothetis
,
N. K.
(
2009
).
Directed interactions between auditory and superior temporal cortices and their role in sensory integration.
Frontiers in Integrative Neuroscience
,
3
, doi: 10.3389/neuro.07.007.2009.
Kayser
,
C.
,
Petkov
,
C. I.
,
Augath
,
M.
, &
Logothetis
,
N. K.
(
2005
).
Integration of touch and sound in auditory cortex.
Neuron
,
48
,
373
384
.
Kayser
,
C.
,
Petkov
,
C. I.
,
Augath
,
M.
, &
Logothetis
,
N. K.
(
2007
).
Functional imaging reveals visual modulation of specific fields in auditory cortex.
Journal of Neuroscience
,
27
,
1824
1835
.
Kayser
,
C.
,
Petkov
,
C. I.
, &
Logothetis
,
N. K.
(
2008
).
Visual modulation of neurons in auditory cortex.
Cerebral Cortex
,
18
,
1560
1574
.
Kermadi
,
I.
,
Liu
,
Y.
,
Tempini
,
A.
,
Calciati
,
E.
, &
Rouiller
,
E. M.
(
1998
).
Neuronal activity in the primate supplementary motor area and the primary motor cortex in relation to spatio-temporal bimanual coordination.
Somatosensory Motor Research
,
15
,
287
308
.
Komura
,
Y.
,
Tamura
,
R.
,
Uwano
,
T.
,
Nishijo
,
H.
, &
Ono
,
T.
(
2005
).
Auditory thalamus integrates visual inputs into behavioral gains.
Nature Neuroscience
,
8
,
1203
1209
.
Lakatos
,
P.
,
Chen
,
C. M.
,
O'Connell
,
M. N.
,
Mills
,
A.
, &
Schroeder
,
C. E.
(
2007
).
Neuronal oscillations and multisensory interaction in primary auditory cortex.
Neuron
,
53
,
279
292
.
Lamarre
,
Y.
,
Busby
,
L.
, &
Spidalieri
,
G.
(
1983
).
Fast ballistic arm movements triggered by visual, auditory, and somesthetic stimuli in the monkey. I. Activity of precentral cortical neurons.
Journal of Neurophysiology
,
50
,
1343
1358
.
Laurienti
,
P. J.
,
Burdette
,
J. H.
,
Maldjian
,
J. A.
, &
Wallace
,
M. T.
(
2006
).
Enhanced multisensory integration in older adults.
Neurobiology of Aging
,
27
,
1155
1163
.
Ma
,
W. J.
,
Zhou
,
X.
,
Ross
,
L. A.
,
Foxe
,
J. J.
, &
Parra
,
L. C.
(
2009
).
Lip-reading aids word recognition most in moderate noise: A Bayesian explanation using high-dimensional feature space.
PLoS One
,
4
,
e4638
.
Macmillan
,
N.
, &
Creelman
,
C.
(
1991
).
Detection theory: A user's guide.
UK
:
Cambridge University Press
.
Maier
,
J. X.
,
Neuhoff
,
J. G.
,
Logothetis
,
N. K.
, &
Ghazanfar
,
A. A.
(
2004
).
Multisensory integration of looming signals by rhesus monkeys.
Neuron
,
43
,
177
181
.
Martuzzi
,
R.
,
Murray
,
M. M.
,
Michel
,
C. M.
,
Thiran
,
J. P.
,
Maeder
,
P. P.
,
Clarke
,
S.
,
et al
(
2007
).
Multisensory interactions within human primary cortices revealed by BOLD dynamics.
Cerebral Cortex
,
17
,
1672
1679
.
Meredith
,
M. A.
, &
Allman
,
B. L.
(
2009
).
Subthreshold multisensory processing in cat auditory cortex.
NeuroReport
,
20
,
126
131
.
Meredith
,
M. A.
, &
Stein
,
B. E.
(
1986
).
Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration.
Journal of Neurophysiology
,
56
,
640
662
.
Miller
,
J.
(
1982
).
Divided attention: Evidence for coactivation with redundant signals.
Cognitive Psychology
,
14
,
247
279
.
Miller
,
J.
,
Ulrich
,
R.
, &
Lamarre
,
Y.
(
2001
).
Locus of the redundant-signals effect in bimodal divided attention: A neurophysiological analysis.
Perception and Psychophysics
,
63
,
555
562
.
Molholm
,
S.
,
Ritter
,
W.
,
Murray
,
M. M.
,
Javitt
,
D. C.
,
Schroeder
,
C. E.
, &
Foxe
,
J. J.
(
2002
).
Multisensory auditory–visual interactions during early sensory processing in humans: A high-density electrical mapping study.
Brain Research, Cognitive Brain Research
,
14
,
115
128
.
Mordkoff
,
J. T.
,
Miller
,
J.
, &
Roch
,
A. C.
(
1996
).
Absence of coactivation in the motor component: Evidence from psychophysiological measures of target detection.
Journal of Experimental Psychology: Human Perception and Performance
,
22
,
25
41
.
Murray
,
M. M.
,
Foxe
,
J. J.
,
Higgins
,
B. A.
,
Javitt
,
D. C.
, &
Schroeder
,
C. E.
(
2001
).
Visuo-spatial neural response interactions in early cortical processing during a simple reaction time task: A high-density electrical mapping study.
Neuropsychologia
,
39
,
828
844
.
Neil
,
P. A.
,
Chee-Ruiter
,
C.
,
Scheier
,
C.
,
Lewkowicz
,
D. J.
, &
Shimojo
,
S.
(
2006
).
Development of multisensory spatial integration and perception in humans.
Developmental Science
,
9
,
454
464
.
Nickerson
,
R. S.
(
1973
).
Intersensory facilitation of reaction time: Energy summation or preparation enhancement?
Psychological Review
,
80
,
489
509
.
Perrault
,
T. J.
, Jr.,
Vaughan
,
J. W.
,
Stein
,
B. E.
, &
Wallace
,
M. T.
(
2005
).
Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli.
Journal of Neurophysiology
,
93
,
2575
2586
.
Porter
,
K. K.
,
Metzger
,
R. R.
, &
Groh
,
J. M.
(
2006
).
Representation of eye position in primate inferior colliculus.
Journal of Neurophysiology
,
95
,
1826
1842
.
Porter
,
K. K.
,
Metzger
,
R. R.
, &
Groh
,
J. M.
(
2007
).
Visual- and saccade-related signals in the primate inferior colliculus.
Proceedings of the National Academy of Sciences, U.S.A.
,
104
,
17855
17860
.
Raab
,
D. H.
(
1962
).
Statistical facilitation of simple reaction times.
Transactions of the New York Academy of Sciences
,
24
,
574
590
.
Rach
,
S.
, &
Diederich
,
A.
(
2006
).
Visual–tactile integration: Does stimulus duration influence the relative amount of response enhancement?
Experimental Brain Research
,
173
,
514
520
.
Rockland
,
K. S.
, &
Ojima
,
H.
(
2003
).
Multisensory convergence in calcarine visual areas in macaque monkey.
International Journal of Psychophysiology
,
50
,
19
26
.
Romanski
,
L. M.
(
2007
).
Representation and integration of auditory and visual stimuli in the primate ventral lateral prefrontal cortex.
Cerebral Cortex
,
17(Suppl. 1)
,
i61
i69
.
Romei
,
V.
,
Murray
,
M. M.
,
Cappe
,
C. C.
, &
Thut
,
G.
(
2009
).
Preperceptual and stimulus-selective enhancement of low-level human visual cortex excitability by sounds.
Current Biology
,
19
,
1799
1805
.
Romei
,
V.
,
Murray
,
M. M.
,
Merabet
,
L. B.
, &
Thut
,
G.
(
2007
).
Occipital transcranial magnetic stimulation has opposing effects on visual and auditory stimulus detection: Implications for multisensory interactions.
Journal of Neuroscience
,
27
,
11465
11472
.
Ross
,
L. A.
,
Saint-Amour
,
D.
,
Leavitt
,
V. M.
,
Javitt
,
D. C.
, &
Foxe
,
J. J.
(
2007
).
Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments.
Cerebral Cortex
,
17
,
1147
1153
.
Ross
,
L. A.
,
Saint-Amour
,
D.
,
Leavitt
,
V. M.
,
Molholm
,
S.
,
Javitt
,
D. C.
, &
Foxe
,
J. J.
(
2007
).
Impaired multisensory processing in schizophrenia: Deficits in the visual enhancement of speech comprehension under noisy environmental conditions.
Schizophrenia Research
,
97
,
173
183
.
Rouger
,
J.
,
Lagleyre
,
S.
,
Fraysse
,
B.
,
Deneve
,
S.
,
Deguine
,
O.
, &
Barone
,
P.
(
2007
).
Evidence that cochlear-implanted deaf patients are better multisensory integrators.
Proceedings of the National Academy of Sciences, U.S.A.
,
104
,
7295
7300
.
Schlack
,
A.
,
Sterbing-D'Angelo
,
S. J.
,
Hartung
,
K.
,
Hoffmann
,
K. P.
, &
Bremmer
,
F.
(
2005
).
Multisensory space representations in the macaque ventral intraparietal area.
Journal of Neuroscience
,
25
,
4616
4625
.
Schroeder
,
C. E.
, &
Foxe
,
J. J.
(
2002
).
The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex.
Brain Research, Cognitive Brain Research
,
14
,
187
198
.
Schroeder
,
C. E.
,
Lakatos
,
P.
,
Kajikawa
,
Y.
,
Partan
,
S.
, &
Puce
,
A.
(
2008
).
Neuronal oscillations and visual amplification of speech.
Trends in Cognitive Science
,
12
,
106
113
.
Schroeder
,
C. E.
,
Lindsley
,
R. W.
,
Specht
,
C.
,
Marcovici
,
A.
,
Smiley
,
J. F.
, &
Javitt
,
D. C.
(
2001
).
Somatosensory input to auditory association cortex in the macaque monkey.
Journal of Neurophysiology
,
85
,
1322
1327
.
Senkowski
,
D.
,
Molholm
,
S.
,
Gomez-Ramirez
,
M.
, &
Foxe
,
J. J.
(
2006
).
Oscillatory beta activity predicts response speed during a multisensory audiovisual reaction time task: A high-density electrical mapping study.
Cerebral Cortex
,
16
,
1556
1565
.
Smiley
,
J. F.
, &
Falchier
,
A.
(
2009
).
Multisensory connections of monkey auditory cerebral cortex.
Hearing Research
, doi: 10.1016/j.heares.2009.06.019.
Sperdin
,
H.
,
Cappe
,
C.
,
Foxe
,
J. J.
, &
Murray
,
M. M.
(
2009
).
Early, low-level auditorysomatosensory multisensory interactions impact reaction time speed.
Frontiers in Integrative Neuroscience
,
3
, doi:10.3389/neuro.07.002.2009.
Stein
,
B. E.
,
London
,
N.
,
Wilkinson
,
L. K.
, &
Price
,
D. D.
(
1996
).
Enhancement of perceived visual intensity by auditory stimuli: A psychophysical analysis.
Journal of Cognitive Neuroscience
,
8
,
497
506
.
Stein
,
B. E.
, &
Meredith
,
M. A.
(
1993
).
The merging of the senses.
Cambridge, MA
:
The MIT Press
.
Stein
,
B. E.
, &
Stanford
,
T. R.
(
2008
).
Multisensory integration: Current issues from the perspective of the single neuron.
Nature Review Neuroscience
,
9
,
255
266
.
Sugihara
,
T.
,
Diltz
,
M. D.
,
Averbeck
,
B. B.
, &
Romanski
,
L. M.
(
2006
).
Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex.
Journal of Neuroscience
,
26
,
11138
11147
.
Sumby
,
W.
, &
Pollack
,
I.
(
1954
).
Visual contribution to speech intelligibility in noise.
Journal of the Acoustical Society of America
,
26
,
212
215
.
Todd
,
J. W.
(
1912
).
Reaction to multiple stimuli.
Archives of Psychology
,
3
,
1
65
.
Vatikiotis-Bateson
,
E.
,
Eigsti
,
I. M.
,
Yano
,
S.
, &
Munhall
,
K. G.
(
1998
).
Eye movement of perceivers during audiovisual speech perception.
Perception & Psychophysics
,
60
,
926
940
.
Wallace
,
M. T.
,
Carriere
,
B. N.
,
Perrault
,
T. J.
, Jr.,
Vaughan
,
J. W.
, &
Stein
,
B. E.
(
2006
).
The development of cortical multisensory integration.
Journal of Neuroscience
,
26
,
11844
11849
.
Wang
,
Y.
,
Celebrini
,
S.
,
Trotter
,
Y.
, &
Barone
,
P.
(
2008
).
Visuo-auditory interactions in the primary visual cortex of the behaving monkey: Electrophysiological evidence.
BMC Neuroscience
,
9
,
79
.
Welch
,
R. B.
, &
Warren
,
D. H.
(
1986
).
Intersensory interactions.
In J. P. Thomas (Ed.),
Handbook of perception and human performance, volume I: Sensory processes and perception
(pp.
25-21
25-36
).
New York
:
Wiley
.
Werner-Reiss
,
U.
,
Kelly
,
K. A.
,
Trause
,
A. S.
,
Underhill
,
A. M.
, &
Groh
,
J. M.
(
2003
).
Eye position affects activity in primary auditory cortex of primates.
Current Biology
,
13
,
554
562
.