When a single tactile stimulus is presented together with two tones, participants often report perceiving two touches. It is a matter of debate whether this cross-modal effect of audition on touch reflects the interplay between modalities at early perceptual or at later processing stages, and which brain processes determine what in the end is consciously perceived. Event-related brain potentials (ERPs) were recorded while rare single tactile stimuli accompanied by two tones (1T2A) were presented among frequent tactile double stimuli accompanied by two tones (2T2A). Although participants were instructed to ignore the tones and to respond to single tactile stimuli only, they often failed to respond to 1T2A stimuli (“illusory double touches,” 1T2A(i)). ERPs to “illusory double touches” versus “real double touches” (2T2A) differed 50 msec after the (missing) second touch. This suggests that at an early sensory stage, illusory and real touches are processed differently. On the other hand, although similar stimuli elicited a tactile mismatch negativity (MMN) between 100 and 200 msec in a unisensory tactile experiment, no MMN was observed for the 1T2A(i) stimuli in the multisensory experiment. “Tactile awareness” was associated with a negativity at 250 msec, which was enhanced in response to correctly identified deviants as compared to physically identical deviants that elicited an illusion. Thus, auditory stimuli seem to alter neural mechanisms associated with automatic tactile deviant detection. The present findings contribute to the debate of which processing step in the brain determines what is consciously perceived.
In everyday life, information from different senses is combined to enhance perception and to facilitate action control. Reaction times are faster to multisensory stimuli than to unisensory stimuli (Miller, 1991), seeing lip movements enhances auditory speech perception (Sumby & Pollack, 1954), and both multisensory object localization (Stein, Meredith, Huneycutt, & McDade, 1989) and recognition (Amedi, von Kriegstein, van Atteveldt, Beauchamp, & Naumer, 2005; Giard & Peronnet, 1999) are more precise than unisensory judgments. Neurophysiological studies both in animals and humans have identified several subcortical and cortical multisensory convergent sites for all major sensory systems (reviewed, e.g., in Fort & Giard, 2004; Stein, Jiang, & Stanford, 2004; Schroeder et al., 2003). These studies suggest that integration of input from different modalities occurs at various stages of the sensory processing hierarchy and that both bottom–up and top–down connections are involved. Multisensory illusions provide a promising means for investigating how these different brain systems contribute to the final perception of the environment because they induce percepts in the absence of any real event. Prominent examples are the McGurk effect (McGurk & MacDonald, 1976), the ventriloquism effect (Bertelson & Aschersleben, 1998), or the size–weight illusion (Flanagan & Beltzner, 2000).
Shams, Kamitani, and Shimojo (2000) reported a striking multisensory illusion:1 Participants often perceived a single visual flash accompanied by two tones as two flashes. The so called double flash illusion was observed even though participants were asked to ignore the tones. Subsequent studies have demonstrated a similar illusion for visual–tactile stimuli (Bresciani, Dammeier, & Ernst, 2006; Violentyev, Shimojo, & Shams, 2005), tactile–auditory stimuli (Bresciani et al., 2005; Hötting & Röder, 2004), and sensorimotor–visual stimuli (Kunde & Kiesel, 2006). The illusion has been shown to be highly reliable provided that stimuli of both modalities were presented in spatial and temporal proximity.
Moreover, prior information about the illusion did not affect its likelihood (Shams et al., 2000). Therefore, it has been concluded that the double flash illusion is due to cross-modal interactions at sensory processing stages rather than to cognitive or attentional processes (Shams, Kamitani, & Shimojo, 2002). Additional support for this interpretation comes from neuroimaging studies. For the double flash illusion, enhanced activity in primary visual cortex was observed when a flash was accompanied by tones (Zhang & Chen, 2006). This neuronal activity appeared to be enhanced when the tones induced an illusory visual perception (Watkins, Shams, Tanaka, Haynes, & Rees, 2006). These results point toward an influence of auditory input on early visual processing stages. However, the temporal resolution of fMRI is insufficient to provide precise information about the timing of multisensory interactions. For example, an alternative explanation could be that the changes in visual cortex activation reflect back-projections from “higher” multisensory areas rather than bottom–up driven activity.
Event-related brain potentials (ERPs) allow a more precise disentangling of neural systems involved in perceptual–cognitive tasks. Shams, Iwaki, Chawla, and Bhattacharya (2005) and Shams, Kamitani, Thompson, and Shimojo (2001) used an ERP approach to study the double flash illusion. They summed ERPs to a double beep and ERPs to a single visual flash, and subtracted the result from ERPs to a multisensory stimulus that consisted of a double beep and a single visual flash (multisensory stimuli associated with an illusory percept of two flashes). They found an occipital difference wave diverging from zero as early as 40–100 msec after stimulus onset. Thus, they concluded that sounds modulate early visual cortex activity. Recently, the subtraction approach has been criticized as activity common to all conditions, such as stimulus expectation, stimulus evaluation, or motor preparation and execution, is added once but subtracted twice (Gondan, Niederhaus, Rösler, & Röder, 2005; Teder-Sälejärvi, McDonald, Di Russo, & Hillyard, 2002). To remedy this, a no-stimulus condition has been added to the bimodal ERPs in a recent study by Mishra, Martinez, Sejnowski, and Hillyard (2007). This procedure controls for common stimulus preceding slow waves that might, if subtracted twice, produce artificial multisensory ERP effects. Mishra et al. (2007) modeled the sources of the ERP difference waves and found a complex pattern of early cross-modal interactions (e.g., 20–60 msec after the second sound) in auditory and visual cortices as well as subsequent activity in superior temporal polymodal areas. A trial-by-trial analysis showed that activity of primary auditory cortex correlated with participants' subjective perception. Participants who were more likely to see illusory flashes could be distinguished from participants who were less likely to see illusory flashes based on multisensory interactions in visual cortex.
The present study used an oddball approach to investigate the impact of auditory stimuli on tactile perception. The mismatch negativity (MMN) commonly observed in such tasks has mainly been investigated in the auditory domain. The MMN is elicited by rare deviant stimuli in a train of homogenous stimuli with a latency of 100–250 msec after stimulus onset. The predominant neural generators of the auditory MMN have been localized in auditory cortex (for reviews, see Kujala, Tervaniemi, & Schröger, 2007; Näätänen, Tervaniemi, Sussman, Paavilainen, & Winkler, 2001; Näätänen, 1995). Because the MMN is elicited even in conditions in which participants do not pay attention to the stimulus stream, it has been associated with preattentive sensory memory mechanisms. Subsequent attention switches are indicated by negative (N2b) and positive (P3) deflections in the deviant ERPs (Näätänen, 1992). More recently, MMN-like ERP effects have been reported for the visual (Czigler, Weisz, & Winkler, 2006; Pazo-Alvarez, Cadaveira, & Amenedo, 2003) as well as for the tactile modality (Akatsuka et al., 2005; Shinozaki, Yabe, Sutoh, Hiruma, & Kaneko, 1998; Kekoni et al., 1997).
In the context of multisensory perception, the auditory MMN has been used to test whether nonauditory stimuli are capable of altering auditory processing. For example, when the auditory syllable “ba” is presented while lips articulating “ga” are seen, people often perceive “da,” a mixture of both syllables (McGurk & MacDonald, 1976). When the same auditory syllable was presented together with a frequent congruent and a rare incongruent visual stimulus (lip movements), an auditory MMN was elicited by the latter although there was no physical auditory change (Saint-Amour, De Sanctis, Molholm, Ritter, & Foxe, 2007; Colin, Radeau, Soquet, Dachy, & Deltenre, 2004; Sams et al., 1991). Dipole analyses have localized the neural generators of this “multisensory MMN” in auditory cortex. Similarly, an MMN was elicited by an illusory shift of the sound's location induced by spatially discrepant visual events (Stekelenburg, Vroomen, & de Gelder, 2004; Colin et al., 2002). Both the scalp distribution and the time course of this “multisensory” MMN were very similar to the auditory MMN. This was interpreted as evidence for the capacity of nonauditory input to trigger deviant responses in the auditory system. Because the auditory MMN has been associated with sensory memory, it has been suggested that these cross-modal effects probably take place on the stage of auditory sensory memory (Stekelenburg et al., 2004). Unfortunately, the EEG studies cited so far did not record behavioral indices demonstrating the participants' percept in illusion trials. Therefore, it was not possible to assess to which extent the MMN varied with participants' perception of the critical multisensory stimuli.
The present study explored the impact of auditory stimuli on deviant detection in the tactile system and linked ERP data to participants' subjective perception. We conducted a unisensory tactile experiment in order to assess deviant-related ERPs in the tactile system. Double tactile stimuli were presented as standard stimuli (2T) and single tactile stimuli as task-irrelevant deviants (1T). Participants were asked to respond to tactile triple stimuli (3T) only. ERPs to 2T standard stimuli were compared to ERPs to 1T deviant stimuli. Based on previous reports on tactile deviant processing (Kekoni et al., 1997), we predicted an enhanced negativity to deviant stimuli in a time range between 100 and 300 msec after deviant onset, which was expected to be followed by a slow positive wave.
In the multisensory experiment, a similar oddball design was used: As frequent standard stimuli, two tactile stimuli were delivered together with two tones (2T2A). Additionally, three types of deviants were introduced: one touch together with two tones (1T2A), one touch together with one tone (1T1A), and two touches accompanied by one tone (2T1A; see Figure 1 for an overview of all experimental conditions). Participants were asked to ignore the tones and to respond to single tactile stimuli only. As has been shown recently, a combination of a single tactile stimulus with two tones (1T2A) often elicits the illusory perception of two touches (Bresciani et al., 2005; Hötting & Röder, 2004). If a participant perceived the 1T2A stimulus as one touch [veridical perception, 1T2A(v)], she was expected to respond. In contrast, if a participant perceived the 1T2A stimulus as two touches [illusory perception, 1T2A(i)], she was expected to miss the deviant, that is, to withhold the response. The latter were the critical trials for the ERP analysis. The key question of the present study was whether the illusion trials elicited a mismatch response and subsequent ERPs that were similar to those observed in the unisensory tactile experiment. Auditory stimulation did not differ between standards and the critical 1T2A deviants in the multisensory experiment. Both in the multisensory and in the tactile experiment, standard stimuli comprised two tactile stimuli and single tactile stimuli were always deviant events. Therefore, it was possible to attribute differences in deviant-related ERPs between the unisensory tactile and multisensory experiment to auditory–tactile interactions.
Since earlier studies have shown that tones do not evoke an illusory percept of two touches in every trial in all participants, neural correlates of “tactile awareness” could be assessed by comparing ERPs to 1T2A stimuli which elicited an illusory percept of two tactile stimuli [1T2A(i)] and the same physical stimuli which did not elicit an illusory percept of two touches [1T2A(v)], respectively.
UNISENSORY TACTILE EXPERIMENT
Twenty-one students from the University of Hamburg took part in the experiment. One participant was excluded due to excessive eye movement artifacts. Four participants did not perceive an illusion in the multisensory experiment in a sufficient number of trials, that is, they mostly responded to the 1T2A stimuli. According to our criterion, more than 30 artifact-free ERP trials to 1T2A stimuli without response were needed to average an ERP to illusion trials. Data for these participants were disregarded. Thus, the final sample comprised 16 participants (2 men, 15 right-handed, mean age = 29 years, range = 19–40 years). All participants were naïve regarding the purpose of the experiment. They reported normal hearing and touch sensitivity and no history of neurological diseases. Participants gave written informed consent; the experiment was performed in accordance with the ethical standards laid down in the Declaration of Helsinki (2000). Participants received course credits or monetary compensation for taking part.
Blindfolded participants were seated in an electrically shielded and sound-attenuating room. Tactile stimuli were delivered by a custom-made tactile device consisting of a relay controlled by a personal computer. Whenever a signal from the computer triggered the relay, a metallic pin (0.8 mm in diameter) moved up by 0.35 mm orthogonal to the skin surface, eliciting the perception of a light touch. The duration of a single tactile stimulus was 20 msec. Double and triple tactile stimuli were produced by presenting two and three tactile stimuli, respectively, separated by an SOA of 100 msec. Participants placed their right index finger on the tactile device, which was put into a kitchen mitten to attenuate the faint noise emerging when the tactile device was operated. Additionally, white noise [47 dB (A)] was presented via headphones to mask any residual sounds produced by the tactile stimulators. The tactile device was placed on a table 45 cm away from the participant and approximately 15° to the right of the body midline.
There were three different trial types (see Figure 1, top): tactile double stimuli (2T, standards, 80% of all trials, n = 900), tactile single stimuli (1T, deviants, 10% of all trials, n = 111), and tactile triple stimuli (3T, targets, 10% of all trials, n = 111). Trials were divided into three blocks. Within each block, stimuli were presented in a random order with a randomly varying intertrial interval (1000–1200 msec). Participants were asked to respond as fast as possible to triple stimuli by lifting their left index finger out of a light gate. Thus, no overt response was expected to single and double tactile stimuli which made them comparable to illusion and standard trials in the multisensory experiment (see below, Multisensory Experiment).
The electroencephalogram (EEG) was recorded from 60 scalp electrodes mounted with equal distance into an elastic cap (Easy Cap, FMS, Herrsching-Breitbrunn, Germany). Vertical eye movements were measured with two electrodes placed under the eyes. Horizontal eye movements were monitored with two electrodes attached to the outer canthi of the eyes. All electrodes were referenced to the nose. Off-line, an averaged right/left mastoid reference was calculated. To avoid eye movements, participants closed their eyes under a sleeping mask. Electrode impedance was kept below 10 kΩ. The bandpass of the amplifiers (Synamps amplifiers; Neuroscan, Sipplingen, Germany) was set to 0.05–100 Hz and the digitization rate was 500 Hz.
Correct responses to target stimuli (3T), false alarms to standard stimuli (2T), and false alarms to deviant stimuli (1T) were counted. Signal detection theory analysis was applied to assess both the sensitivity (d′) to tactile stimuli and the response bias (c) of participants. The correct identification of triple stimuli was counted as “hit” and responses to both single and double tactile stimuli were counted as “false alarms.” d′ for tactile discrimination performance was defined as z(p(hit)) − z(p(false alarms)) and c was defined as 0.5(z(p(hit)) + z(p(false alarms))), where z scores were derived from the inverse cumulative normal distribution (MacMillan & Creelman, 1991).
Hundred milliseconds were subtracted from reaction times to 3T target stimuli to compensate for different time points when targets started to differ from nontarget stimuli in the tactile and multisensory experiment (200 and 100 msec after stimulus onset, respectively; see Methods of the Multisensory Experiment). This allows a comparison of reaction times across experiments. Only reaction times between 150 and 1200 msec were considered.
ERPs were averaged time-locked to the onset of the first tactile stimulus, separately for each participant and condition (see Figure 1, top). All ERPs were referred to a 100-msec prestimulus baseline. The onset of the second tactile stimulus, the time point when deviants started to differ from standards, is marked as time point zero in the figures (see dotted vertical line in Figure 2). Thus, when we talk about “deviant onset,” we mean the time point when standards and deviants started to differ. In seven participants, one to two electrodes with recording artifacts were replaced by a linear interpolation of adjacent electrodes (Picton et al., 2000). Trials with biological artifacts (due to eye and body movements) were removed. Only trials without a behavioral response were considered. Three adjacent electrodes were combined to an electrode cluster to increase the signal-to-noise ratio (see Figure 2).
Time windows for statistical analyses were selected according to results of previous studies on somatosensory deviant ERPs (Kekoni et al., 1997; Kekoni, Hamalainen, McCloud, Reinikainen, & Näätänen, 1996) and according to visual inspection of the standard versus deviant waves in the grand-average ERPs. Mean amplitudes were calculated for time windows 40–90 msec, 100–200 msec, 250–350 msec, and 400–600 msec.
Mean amplitudes were analyzed with an ANOVA comprising three repeated measurement factors: Stimulus (standard vs. deviant), Hemisphere (left vs. right), and Cluster (1–7).2 Whenever an interaction with the factor Stimulus reached significance, separate ANOVAs for single clusters or hemispheres were run. Post hoc, the Stimulus effect for single clusters was tested with paired t tests.
Statistics were computed with the program package SPSS, subroutine GLM for repeated measurements. Huynh/Feldt-corrected p values are reported when appropriate. Post hoc tests for single clusters were calculated with one-tailed paired t tests for time windows with an a priori hypothesis about the polarity of the effect (e.g., deviant-related negativity between 100 and 350 msec and the expected positivity to deviants in the time interval 400–600 msec). In all other cases, two-tailed t tests were applied.
Participants correctly identified 61% [standard error (SE) = 4.6] of the target stimuli (3T) with a mean reaction time of 585 msec (SE = 18). They committed 1.1% (SE = 0.5) false alarms to single tactile deviant stimuli (1T) and 6.4% (SE = 1.5) false alarms to standard stimuli (2T). The average d′ for detecting tactile triple stimuli among single and double tactile stimuli was 2.1 (SE = 0.18) and the response criterion c was 0.76 (SE = 0.12), that is, participants showed a tendency to withhold a response.
ERPs to tactile stimuli are seen in Figure 2. Deviant stimuli (1T) elicited a more positive-going ERP than standard stimuli (2T) between 40 and 90 msec after deviant onset (marked as P50 in Figure 2). This difference was confirmed by a significant Stimulus × Hemisphere × Cluster interaction [F(6, 90) = 3.03, p = .047]. Follow-up analyses showed that the effect was most pronounced over the left frontal scalp [Figure 3; main effect stimulus for Cluster 1 F(1, 15) = 4.70, p = .047; t test for Cluster L1 t(15) = −2.3, p = .034].
Between 100 and 200 msec, a deviant-related negativity was seen at frontal sites [Stimulus × Cluster, F(6, 90) = 3.51, p = .032; the t test was significant for Cluster R1 t(15) = 1.90, p = .033]. This negativity was followed by a slow positive wave to deviant stimuli, which was most pronounced at frontal and central clusters of both hemispheres. This positivity (250–350 msec) was statistically confirmed by a significant Stimulus × Hemisphere interaction [F(1, 15) = 7.47, p = .015] and a significant Stimulus × Cluster interaction [F(6, 90) = 15.75, p < .001]. Follow-up ANOVAs for each hemisphere showed a significant effect of Stimulus for the left and right hemisphere [left hemisphere: Stimulus × Cluster, F(6, 90) = 16.34, p < .001, t tests for single clusters, all t < −2.9, all p < .02 for L1, L2, L3, L4, L5; right hemisphere: Stimulus × Cluster, F(6, 90) = 12.00, p < .001, t tests, all t < −2.5, p < .02 for R1, R2, R3, R4]. The larger positivity for deviant stimuli than for standard stimuli prevailed until the later time epoch between 400 and 600 msec [main effect Stimulus: F(1, 15) = 7.34, p = .016; t tests significant for Clusters L2, L3, L4, L5, R2, R3, R4, R5, R6, all t(15) < −1.9, all p < .04].
Summary and Discussion
Reliable differences between ERPs elicited by tactile deviant and standard stimuli were found. ERPs to deviant stimuli were more positive-going between 40 and 90 msec as compared to standard stimuli and, at frontal sites, more negative-going between 100 and 200 msec. This negativity was followed by a slow positive wave for deviants. The early difference between single touches (1T, deviant stimuli) and double touches (2T, standard stimuli) around 50 msec after deviant onset could reflect either a real mismatch response or a second N80 elicited by the second touch of the standard. Previous ERP studies on tactile deviant detection reported ERP differences between deviants and standards within the first 100 msec after deviant onset. Some authors have reported an enhanced positivity for deviants as compared to standards with a latency of 40–60 msec (Kekoni et al., 1997), whereas others did not find this effect but observed an enhanced negativity between 60 and 90 msec (Restuccia, Marca, Valeriani, Leggio, & Molinari, 2007). Polarity differences across studies might be due to differences in recording techniques, and thus, further studies are necessary to characterize the time course and topography of ERPs indicating early tactile deviant detection. Alternatively, the positive-going ERP to deviants as compared to standard stimuli in the time window between 40 and 90 msec might be due to the difference in tactile stimulation. In the present study, the second tactile stimulus of the standards most likely elicits a second somatosensory N80. The contralateral distribution of this effect supports the latter hypothesis as the N80 is known to have a predominant contralateral scalp distribution.
In the present study, we observed a deviant-related frontally distributed negativity between 100 and 200 msec after deviant onset. Time course and topography of this effect were in accordance with somatosensory mismatch responses reported earlier in active and passive oddball tasks (Restuccia et al., 2007; Kida, Nishihira, Wasaka, Nakata, & Sakamoto, 2004; Kekoni et al., 1996, 1997). In the following, we refer to this effect as tactile MMN. The tactile MMN was followed by a positive wave with a frontal topography that resembles the P3a. This positivity is typically observed to deviant stimuli which do not require a response (Polich, 2004; Yamaguchi & Knight, 1991).
Participants and Procedure
The same participants took part in the tactile and multisensory experiments. The multisensory experiment was always run before the tactile experiment to reduce possible initial training effects for tactile stimuli that might have reduced the susceptibility to the auditory–tactile illusion (Hötting & Röder, 2004). To familiarize participants with the task, a short practice session with 80 tactile trials was presented first.
Tactile stimuli and their timing were the same as in the unisensory tactile experiment. Additional auditory stimuli were presented from a speaker positioned in front of the tactile stimulator [sinusoidal tones, 2100 Hz, 77 dB(A)]. The duration of a single auditory stimulus was 10 msec. Double auditory stimuli were produced by presenting two tones separated by an SOA of 100 msec. In each trial, the first auditory stimulus was presented 25 msec prior to the first tactile stimulus. A previous behavioral study had shown that this timing of auditory and tactile stimuli reliably elicited the “double touch” illusion (Hötting & Röder, 2004). The intertrial interval varied randomly between 1000 and 1200 msec.
There were four different trial types (see Figure 1, bottom): (1) two tactile stimuli combined with two tones (2T2A, standards, 80% of all trials, n = 1800), (2) one tactile stimulus combined with two tones (1T2A, deviants, 6.7% of all trials, n = 150), (3) one tactile stimulus combined with one tone (1T1A, targets, 6.7% of all trials, n = 150), and (4) two tactile stimuli combined with one tone (2T1A, deviants, 6.7% of all trials, n = 150). Trials were presented in a randomized order and grouped into six blocks.
The participants' task was to respond as fast as possible to stimuli comprising a single tactile stimulus. They had to lift their left index finger out of a light gate. Participants were told that tones were task-irrelevant. A missing response to 1T2A stimuli was interpreted as evidence for an illusory perception of two touches when a single tactile stimulus was accompanied by two tones [1T2A(i)]. Thus, both the presumably similar perceived 2T2A standard trials and 1T2A(i) “deviant trials” were not confounded by brain activity related to response preparation or execution (see Data Analysis section for details). The relative number of standard stimuli was matched across the unisensory tactile and multisensory experiment.
EEG recording details were the same as in the unisensory tactile experiment.
False alarms to standard stimuli (2T2A), correct responses to target stimuli (1T1A) and responses to deviant stimuli (1T2A or 2T1A) were counted. d′ was calculated for detecting single tactile stimuli among double tactile stimuli in the multisensory experiment. Therefore, responses to 1T1A and to 1T2A stimuli were counted as “hits” and responses to 2T2A and to 2T1A were counted as “false alarms.” Moreover, the response criterion c was derived to assess participants' response bias.
ERPs were averaged to the first tactile stimulus in each trial, separately for each participant and condition (see Figure 1, bottom). Artifact rejection criteria, baseline correction, and electrode clustering were the same as in the unisensory tactile experiment.3 Time windows for statistical analyses were selected according to the results of the unisensory tactile experiment and based on a visual inspection of the deviant versus standard grand-average ERPs in the multisensory experiment. The first difference between standard and deviant stimuli in the multisensory experiment was a brief modulation of a positive peak at about 50 msec after deviant onset. Due to the transient nature of this ERP and because of slight variations in peak onset across participants, we assessed this effect with a peak amplitude measure of the maximal positive peak within the first 50 msec. Mean amplitudes were calculated for time windows 100–200 msec, 200–250 msec, 250–350 msec, and 400–600 msec.
Two separate analyses were carried out. First, ERPs to multisensory deviant stimuli [1T2A(i)] were compared to ERPs to standard stimuli (2T2A) with an ANOVA comprising three repeated measurement factors: Stimulus (standard vs. deviant), Hemisphere (left vs. right), and Cluster (1–7). Only trials without response were considered. Thus, ERPs to deviant stimuli presumably reflect trials which elicited the illusory perception of two tactile stimuli. Second, ERP differences to physically identical stimuli but with an overt response, and thus, a presumably different subjective perception [1T2A(v)] were compared to ERPs in illusion trials [1T2A(i)]. This analysis was conducted for a subgroup of nine participants who had at least 30 artifact-free trials for both 1T2A(v) and 1T2A(i) stimuli. An ANOVA with the within-participant factors Perception [1T2A(v) vs. 1T2A(i)], Hemisphere (left vs. right), and Cluster (1–7) was run to test correlates of subjective perception.
Finally, to directly compare deviant effects between the multisensory and the unisensory tactile experiment, data of both experiments were entered into an ANOVA comprising four repeated measurement factors: Experiment (unisensory vs. multisensory), Stimulus (standard vs. deviant), Hemisphere (left vs. right), and Cluster (1–7).
Participants committed 5.1% (SE = 1.5) false alarms to multisensory standard stimuli (2T2A) and correctly identified 81% (SE = 4.6) of multisensory targets (1T1A). Mean reaction time for the correct detection of the target stimuli was 552 msec (SE = 23). Participants missed 73% (SE = 5.8) of the 1T2A deviant stimuli. Thus, they presumably perceived a single tactile stimulus accompanied by two tones as two tactile stimuli in most of the trials. Furthermore, participants responded to 61% (SE = 8.6) of the 2T1A deviant stimuli, suggesting that they perceived two tactile stimuli accompanied by one tone as one tactile stimulus in more than half of the trials. ERPs to 2T1A stimuli were not analyzed because both auditory stimulation and motor-related activity differed from standards.
d′ was calculated for detecting a single tactile stimulus among tactile double stimuli in the multisensory experiment. Responses to 1T1A stimuli and responses to 1T2A stimuli were counted as hits, responses to 2T1A stimuli and to 2T2A stimuli were considered as false alarms. The average d′ was 1.52 (SE = 0.12) and the response criterion c was 0.65 (SE = 0.07).
Although d′ was higher in the tactile than in the multisensory experiment [t(15) = 4.14, p = .001], the response criterion did not differ between experiments (p > .2). A direct comparison of behavioral data between experiments, however, has to be interpreted with caution as participants had to detect single tactile stimuli in the multisensory experiment and triple tactile stimuli in the tactile experiment.
ERPs elicited by multisensory standard stimuli (2T2A) and multisensory deviant stimuli [1T2A(i)] are shown in Figure 4.4 The main research question of the present study was whether ERPs to multisensory deviant stimuli perceived as comprising two touches [1T2A(i)] differed from ERPs to standard stimuli (2T2A). This comparison contrasts ERPs to physically different stimuli which elicited subjectively the same percept. Thus, only stimuli without an overt response were included in this analysis. Moreover, ERP correlates of conscious deviant detection were assessed by comparing ERPs to 1T2A stimuli when they were detected [1T2A(v)] and when they were not detected [1T2A(i); Figure 5 4]. In the following, we report these two main lines of analyses: (a) physically differing stimulation–same reported perception; (b) physical identical stimulation–different reported perception, separately for three time epochs.
Peak Amplitudes at ∼50 msec
The first ERP difference between standard and deviant stimuli in the multisensory experiment was observed 50 msec after deviant onset: The positive peak to 1T2A(i) deviant stimuli at 50 msec after deviant onset (P50) was more pronounced than to 2T2A standard stimuli [main effect Stimulus: F(1, 15) = 5.40, p = .035]. This ERP difference was maximal at fronto-central electrode sites [t tests; t(15) < −2, p < .05 for Clusters L2, R1, R3; Figure 4, insert]. The comparison between ERPs to 2T2A standard stimuli and ERPs to 1T2A(i) deviant stimuli for the subgroup of participants who perceived the illusion in at least 30 trials and correctly identified the single tactile stimuli in at least 30 of the 1T2A trials confirmed an enhanced P50 to deviant stimuli (see Figure 5; Table 1 shows the statistics for this subgroup).
|Peak Amplitude (∼50 msec)|
|Mean Amplitude (200–250 msec)|
|Mean Amplitude (250–350 msec)|
|Mean Amplitude (400–600 msec)|
|Stimulus × Hemisphere||ns||ns||ns||ns|
|Stimulus × Cluster||F(6, 48) = 3.41, p = .027||ns||ns||F(6, 48) = 5.87, p = .013|
|Stimulus × Hemisphere × Cluster||ns||ns||F(6, 48) = 4.03, p = .043||F(6, 48) = 5.45, p = .022|
|t tests for single clusters, p < .05||L1, R7||ns||L5, R5||L3|
|Peak Amplitude (∼50 msec)|
|Mean Amplitude (200–250 msec)|
|Mean Amplitude (250–350 msec)|
|Mean Amplitude (400–600 msec)|
|Stimulus × Hemisphere||ns||ns||ns||ns|
|Stimulus × Cluster||F(6, 48) = 3.41, p = .027||ns||ns||F(6, 48) = 5.87, p = .013|
|Stimulus × Hemisphere × Cluster||ns||ns||F(6, 48) = 4.03, p = .043||F(6, 48) = 5.45, p = .022|
|t tests for single clusters, p < .05||L1, R7||ns||L5, R5||L3|
Main effects of Hemisphere or Cluster and interactions of Hemisphere × Cluster are not reported.
ns: p > .1.
The amplitude of the P50 correlated with the physical stimulus properties rather than with the participants' perception (see Figure 5, insert): ERPs to both 1T2A(i) and 1T2A(v) stimuli were more positive-going than ERPs to 2T2A stimuli, whereas the difference between the P50 to 1T2A(i) and 1T2A(v) was not significant (p > .05).
Time Epochs 100–200 msec, 200–250 msec, and 250–350 msec
There was no difference between multisensory deviant and standard stimuli in the time window 100–200 msec (Figure 4; p > .1). Starting at about 200 msec, ERPs to deviant stimuli [1T2A(i)] were more negative than ERPs to standard stimuli (2T2A). The earliest part of this deviant-related negativity (200–250 msec) failed to reach significance in the overall ANOVA (p > .1). Between 250 and 350 msec, an enhanced negativity to deviant stimuli [1T2A(i)] as compared to standards (2T2A) resulted in a marginally significant main effect of stimulus [F(1, 15) = 4.52, p = .051] and a marginal significant interaction between Stimulus, Hemisphere, and Cluster [F(6, 90) = 3.15, p = .069] in the overall ANOVA. The difference was most pronounced over the centro-lateral clusters of the right hemisphere [right hemisphere: main effect Stimulus, F(1, 15) = 4.83, p = .044; t tests for Clusters R2, R3, R4, R5: t(15) > 2, p < .03; Figure 6].
ERPs in this time range were sensitive to participants' perception as well (Figure 5). Although 1T2A(i) trials elicited only a relatively small negativity between 200 and 250 msec as compared to standard stimuli, a much more negative-going ERP was elicited by 1T2A(v) stimuli [Perception × Cluster: F(6, 48) = 5.22, p = .018]. The difference between 1T2A(v) and 1T2A(i) trials was maximal over frontal clusters [main effect Perception for Cluster 1: F(1, 8) = 15.55, p = .024; Perception × Hemisphere for Cluster 2: F(1, 8) = 8.37, p = .02; for L1, L2, R1, R2: all t(8) > 2, p < .04; Figure 7].
Time Epoch 400–600 msec
A long-lasting deviant-related negativity was seen over the frontal and lateral scalp when comparing ERPs to 1T2A(i) stimuli to ERPs to 2T2A standard stimuli [Stimulus × Cluster: F(6, 90) = 4.82, p = .023; separate analyses for single clusters: main effect stimulus, F(1, 15) > 4.5, p < .05 for Clusters 1 and 3; t(15) > 2.3, p < .05 for L3 and R3; Figures 4 and 6].
However, when participants detected the multisensory deviant stimuli [1T2A(v)], a positive wave was elicited as compared to illusion trials [Perception × Cluster: F(6, 48) = 5.35, p = .009]. This ERP difference between 1T2A(v) and 1T2A(i) trials had a centro-posterior scalp distribution [main effect Perception for Clusters 3, 4, 5, 6, and 7: all F(1, 8) > 8, p < .03; Perception × Hemisphere for Clusters 4 and 6: all F(1, 8) > 6, p < .04; for L3, L4, L5, L6, L7, R4, R5, R6, R7: all t(8) < −3, p < .01; Figures 5 and 7].
Comparison of Deviant-related Effects between the Unisensory Tactile and Multisensory Experiment
Figure 8 shows deviant minus standard difference waves for the unisensory tactile and multisensory experiments. An early positivity measured with mean amplitudes between 40 and 90 msec was seen in both the unisensory tactile and the multisensory experiment [Stimulus × Hemisphere × Cluster: F(6, 90) = 4.37, p = .006; no significant interaction with the factor Experiment]. The maximum of this deviant effect was over the left (contralateral) hemisphere in both experiments with positive amplitudes over the frontal scalp and negative amplitudes over parietal sites.
In contrast to the unisensory tactile experiment, no effect of stimulus type in the time window 100–200 msec was observed in the multisensory experiment [Experiment × Stimulus × Cluster: F(6, 90) = 4.19, p = .018]. An enhanced negativity to deviant stimuli as compared to standard stimuli [1T2A(i) vs. 2T2A] was seen in a later time window between 250 and 350 msec in the multisensory experiment (see above). In contrast, in the unisensory tactile experiment, ERPs (250–350 msec) to deviants (1T) were more positive-going than to standard stimuli (2T) [Experiment × Stimulus × Cluster: F(6, 90) = 9.83, p = .001].
Late deviant-related effects (400–600 msec) were negative for 1T2A(i) versus 2T2A trials in the multisensory experiment, but positive for 1T versus 2T trials in the unisensory tactile experiment [Experiment × Stimulus × Cluster: F(6, 90) = 8.0, p = .002].
The present study explored neural correlates of cross-modally induced changes in tactile perception. Auditory–tactile stimuli were presented in an oddball task: Rare deviant stimuli, consisting of a single touch and two tones (1T2A), were presented among frequent standard stimuli, consisting of two touches and two tones (2T2A). The participants' task was to respond to any stimulus comprising one tactile stimulus irrespectively of the number of tones. When participants did not respond to 1T2A deviants, we concluded that they perceived a double touch. This was the case in almost three quarters of the 1T2A trials. Deviant-related ERPs starting at 100 msec after deviant onset were significantly influenced by tones. A tactile MMN between 100–200 msec for single touches was observed in a unisensory tactile experiment but was not seen for auditory–tactile illusion stimuli [1T2A(i)]. ERPs to deviants [1T2A(i)] as compared to standards (2T2A) were more negative-going around 250 msec after deviant onset in the multisensory experiment. This effect was larger for veridical trials [1T2A(v)] than for illusion trials [1T2A(i)]. Finally, the detection of 1T2A(v) deviants was associated with a slow positive wave starting at 300 msec. In contrast, physically identical stimuli elicited a long-lasting negative wave over frontal areas when they were not detected as deviants, that is, when the illusion presumably was experienced [1T2A(i)].
In both the unisensory tactile and the multisensory experiment, the first difference between deviants (two touches) and standards (one touch) was seen as a relative positive deflection at around 50 msec with respect to the time point when deviants and standards started to differ physically. This ERP effect was most likely due to additional sensory ERPs elicited by the second tactile stimulus comprising the standards (2T and 2T2A) as compared to single tactile stimuli. The left (contralateral) distribution of this effect in both experiments supports this assumption. The fact that both detected deviants [1T2A(v)] and nondetected deviants [1T2A(i)] were accompanied by such a positive deflection suggests that the illusory perception of two touches does not elicit the same early sensory activity as two real touches.
At first glance, the finding that early deviant-related ERPs were very similar in the tactile and multisensory experiment seems to be at odds with previous EEG and MEG studies reporting auditory–tactile interactions with latencies shorter than 100 msec (Murray et al., 2005; Gobbelé et al., 2003; Foxe et al., 2000). Source localizations of these effects, as well as fMRI studies, have suggested that these interactions take place in low-level sensory cortices (Schürmann, Caetano, Hlushchuk, Jousmaki, & Hari, 2006; Kayser, Petkov, Augath, & Logothetis, 2005; Foxe et al., 2002). Recent results of neurophysiological recordings in animals have demonstrated a convergence of auditory and somatosensory input into posterior auditory association cortices (Schroeder & Foxe, 2002; Schroeder et al., 2001). The timing and the laminar profile of these responses were in line with feedforward inputs. Moreover, direct anatomical connections between the auditory and somatosensory cortices have been reported (Cappe & Barone, 2005). However, there are some methodological differences between the current experiment and previous EEG studies: Previous EEG experiments used simultaneous bimodal auditory–tactile stimulation and compared the ERP response to bimodal stimuli to those of the unimodal responses (often the sum of the unisensory responses). Moreover, in most of those studies, participants performed a simple detection task and there was no conflict between the inputs of the different sensory modalities. The present experiment was not designed to detect the earliest possible signs of auditory–tactile interactions but to test whether tones have an impact on tactile deviant detection and which neural processing stages correlate with participants' perception when a multisensory illusion is elicited. Therefore, tactile and auditory stimuli were presented with a short time lag and participants had to attend to tactile stimuli to detect rare deviants within that modality. In some trials, incongruent auditory and tactile input elicited a perceptual illusion. For ERP analyses, responses to frequent standard stimuli were compared to those to rare deviants in a multisensory and in a unisensory experiment. Thus, our results do not contradict or argue against multisensory interactions with a latency below 100 msec. Our findings suggest that illusory percepts (double touches) elicited by a second modality (tones) do not elicited the same sensory response as veridical percepts (two physically presented touches).
This interpretation of the data is consistent with other recent findings. A unisensory tactile study, for example, showed that early ERPs to detected and undetected masked tactile stimuli did not differ (Schubert, Blankenburg, Lemm, Villringer, & Curio, 2006). Models of consciousness, which are mainly based on experiments using visual stimuli, postulate that awareness is correlated with activity in visual association cortex rather than with activity in primary visual cortex (Koch & Braun, 1996). For the auditory domain, a recent article on the neuronal basis of the ventriloquism illusion reported that early ERP correlates of auditory–visual interactions did not vary with the participants' perception (Bonath et al., 2007). In this study, a visually induced shift in sound localization was linked to a later ERP at 260 msec most likely generated in the planum temporale, which is part of auditory association cortex. For the double flash illusion, however, it has been shown that early activity (20–50 msec) in auditory cortex varied with the participants' visual perception (Mishra et al., 2007). Thus, future research needs to further address the contribution of early bottom–up driven activity in sensory cortex for subjective correlates of multisensory stimulation.
The present results suggest that tones are capable of modulating tactile processing, starting 100 msec after deviant onset. Although a tactile MMN to single touches (1T deviants) among frequent double touches (2T standards) could be reliably demonstrated in the present unisensory tactile experiment, a similar tactile ERP response was not observed for single touches accompanied by double tones [1T2A(i) and 1T2A(v)] in the multisensory experiment. The MMN is thought to be generated by an automatic change-detection process that preattentively matches the actual input with a sensory memory trace (Kujala et al., 2007). This memory trace for somatosensory stimuli seems to be modified by the presence of task-irrelevant tones, irrespectively of the behavioral report of the participants. The fact that the absence of a tactile MMN did not correlate with participants' subjective perception suggests that the influence of tones on tactile sensory representation is automatic and does not determine “tactile awareness.”
The idea of multisensory interplay at the level of sensory memory mechanisms is compatible with previous results from auditory–visual experiments using the auditory MMN to indicate the influence of spatially corresponding and noncorresponding visual stimuli on auditory processing. For example, Colin et al. (2002) reported that an auditory MMN between 130 and 150 msec to spatially deviating tones was suppressed when a visual stimulus was presented at the position of the standard location (ventriloquism effect). Consistently, Stekelenburg et al. (2004) reported that an auditory MMN between 100 and 200 msec poststimulus was elicited when tones were accompanied by spatially deviating visual stimuli. These visual stimuli shifted the perceived localization of the tones away from the location of the frequent auditory stimuli, thus creating spatially deviating sounds. In previous MMN studies on multisensory illusions, participants did not respond to the multisensory stimuli. The subjective percept of the participants was, therefore, indirectly inferred by measuring the MMN response in these studies. In contrast, in the present study, it was possible to assess the subjective percept of the participants more directly based on their overt or withheld response.
Although our participants did not report perceiving two thirds of the deviant stimuli [1T2A(i)], we found evidence for some brain mechanisms that seem to track the veridical difference between standard and deviant stimuli. Illusory stimuli [1T2A(i)] elicited an enhanced centro-lateral negativity starting around 200 msec after deviant onset as compared to standards (2T2A) even though both stimuli appeared to be subjectively perceived the same. The amplitude of this deviant-related negativity for illusion trials [1T2A(i)] was enhanced when participants veridically detected the deviant stimulus [1T2A(v); N200–250]. The time course and the scalp distribution of this effect resemble a negative deviant response observed for participants performing an active tactile oddball task (Kekoni et al., 1996, 1997). Kekoni et al. (1996, 1997) suggested that this negativity corresponds to the auditory N2b reported in analogous auditory experiments. An N2b has been found to be elicited when a deviant is sufficiently salient to catch participants' attention (Näätänen, Simpson, & Loveless, 1982). The auditory N2b has been associated with an automatic classification mechanism (Ritter, Simson, & Vaughan, 1983). One could therefore speculate that a certain activation threshold (indicated by the amplitude of the N200–250) of the neural system has to be reached in order to classify a stimulus as a deviant. The present results suggest that auditory stimuli are capable of modulating this classification system for tactile stimuli. In some trials, where the auditory influence was presumably rather low, sufficient evidence to elicit an N200–250 might have been accumulated. In other trials, the interplay of auditory and tactile stimulation might have resulted in insufficient evidence for a deviant occurrence, resulting in an attenuated N200–250 and a withholding of the response.
Only 1T2A(v) stimuli, that is, stimuli subjectively perceived as deviants, triggered a parietal positive wave with a maximum between 400 and 600 msec after deviant onset. This positivity most likely belongs to the P3 family (Yamaguchi & Knight, 1991) commonly associated with target detection.
In sum, it might be speculated that the N200–250 reflects the activity of a brain mechanism that accumulates evidence for triggering an orienting response and that the output of this system determines what is consciously perceived.
Finally, ERP correlates to illusory double touches [1T2A(i)] and to real double touches (2T2A) differed in a slow negative wave over the frontal scalp starting at 400 msec. It might be speculated that this negativity reflects some preconscious deviant detection processes or the detection of an incongruity between the inputs of two modalities. For the auditory modality, Sams, Paavilainen, Alho, and Näätänen (1985) found an enhanced slow negativity at fronto-central sites for missed deviants at the discrimination threshold.
Alternatively, the frontal ERP effect for the double touch illusion [1T2A(i)] might be a genuine neural correlate of illusory percepts. Two recent fMRI studies reported activations in prefrontal areas when participants experienced a tactile (Blankenburg, Ruff, Deichmann, Rees, & Driver, 2006) or a multisensory illusion (Ehrsson, Spence, & Passingham, 2004). In the study by Blankenburg et al. (2006), two different positions on the upper arm were stimulated in a rapid succession. Participants reported perceiving multiple touches propagating across intervening locations along the arm (“rabbit illusion”). Activity in somatosensory cortex was very similar for these illusory touches as compared to real touches. Only illusion trials, however, were accompanied by prefrontal cortex activation. The authors speculated that prefrontal cortex might be involved in a top–down modulation of early sensory processing in primary somatosensory cortex.
Finally, some possible objections need to be discussed. It could be argued that the participants' responses were determined by a bias toward the tones rather than by a genuine illusory tactile percept. Previous studies varying the timing of auditory and tactile stimuli in an experiment addressing the double touch illusion have, however, provided evidence against a response bias account (Bresciani et al., 2005). Signal detection analysis of the present behavioral data further supports this view: d′ values for the multisensory experiment suggest that participants were able to discriminate single and double tactile stimuli and did not just report the number of tones. Moreover, the tendency to withhold a response was similar in the multisensory and the unisensory tactile experiment. Nevertheless, some responses may reflect errors rather than the participants' perception. As illusion trials were associated with a withholding of a response, incorrect “yes” responses were not considered for the critical 1T2A(i) trials. If withholding a response to 1T2A stimuli, however, was due to the lack of responding rather than to altered stimulus perception, we would expect an orienting response indicated by a P3a (Polich, 2004). Indeed, a P3a was observed in the unisensory tactile experiment to rare deviants not requiring a response (see Figure 2). As seen in Figures 4 and 5, there was no sign of a P3a in the ERPs to 1T2A(i).
Another objection might be that the ERP differences observed between 2T2A and 1T2A trials could predominantly reflect a shift in attention toward the auditory stimuli. This, however, seems to be rather unlikely. Only trials with the same auditory stimulation were considered for ERP analyses in the multisensory experiment. One could further argue that differences between veridical and illusion trials [1T2A(v) vs. 1T2A(i)] might be due to transient shifts in attention toward the tactile stimuli for veridical trials and toward auditory stimuli in the illusion trials. Moreover, these shifts of attention might be associated with differences in anticipatory potentials. Intermodal attention is known to modulate early ERPs around 100 msec (Hötting, Rösler, & Röder, 2003; Talsma & Kok, 2001), and differences in anticipatory prestimulus potentials would be expected to develop from stimulus onset on. Because ERP differences between veridical and illusion trials were not observed before 200 msec, these explanations seem rather unlikely.
Using ERPs, we were able to show that auditory stimuli alter neural mechanisms associated with tactile deviant detection and possibly tactile awareness. Somatosensory processing stages between 100 and 200 msec were modulated by the presence of simultaneously presented tones; the activity of later neural classification mechanism varied with tactile perception. This finding is in line with models of consciousness, assuming that conscious perception depends on the activity of sensory association cortex and feedback connections. Finally, the results of the present study substantiate previous evidence for a frontal brain system that seems to keep track of the real stimulus when an illusory percept occurs.
The study was supported by the Deutsche Forschungsgemeinschaft (DFG, Ro1226/4-3) to B. R. We thank Dagmar Tödter and Sybille Röper for their help with data acquisition and Dr. Margriet Groen and Dr. Thérèse Collins for proof reading.
Reprint requests should be sent to Kirsten Hötting, Biological Psychology and Neuropsychology, University of Hamburg, Von-Melle-Park 11, D-20146 Hamburg, Germany, or via e-mail: email@example.com.
The term “multisensory illusion” is used in the following to characterize the phenomenon that stimuli of one sensory modality can alter the perception of stimuli of another modality.
To test whether clustering had an impact on the results, data were also analyzed with an ANOVA including the factor electrode instead of cluster. These analyses confirmed the statistical reliability of the primary significant effects reported in the Results section.
As the 100-msec prestimulus baseline in the multisensory experiment included ERP responses to the first sound, we referred ERPs to a time interval from −100 to −25 msec relative to the first tactile stimulus in an additional analysis. Because ERPs hardly changed, we report results for the −100 to 0 msec prestimulus baseline only.
In order to test whether different signal-to-noise ratios due to different numbers of trials in the averages for each condition caused the observed ERP differences, we randomly selected the same number of trials for the condition with the larger number of trials as were available in the condition with the lower number of trials. This did not change the pattern of results (see Supplementary Material).