Abstract

Processing of a given target is facilitated when it is defined within the same (e.g., visual–visual), compared to a different (e.g., tactile–visual), perceptual modality as on the previous trial [Spence, C., Nicholls, M., & Driver, J. The cost of expecting events in the wrong sensory modality. Perception & Psychophysics, 63, 330–336, 2001]. The present study was designed to identify electrocortical (EEG) correlates underlying this “modality shift effect.” Participants had to discriminate (via foot pedal responses) the modality of the target stimulus, visual versus tactile (Experiment 1), or respond based on the target-defining features (Experiment 2). Thus, modality changes were associated with response changes in Experiment 1, but dissociated in Experiment 2. Both experiments confirmed previous behavioral findings with slower discrimination times for modality change, relative to repetition, trials. Independently of the target-defining modality, spatial stimulus characteristics, and the motor response, this effect was mirrored by enhanced amplitudes of the anterior N1 component. These findings are explained in terms of a generalized “modality-weighting” account, which extends the “dimension-weighting” account proposed by Found and Müller [Searching for unknown feature targets on more than one dimension: Investigating a “dimension-weighting” account. Perception & Psychophysics, 58, 88–101, 1996] for the visual modality. On this account, the anterior N1 enhancement is assumed to reflect the detection of a modality change and initiation of the readjustment of attentional weight-setting from the old to the new target-defining modality in order to optimize target detection.

INTRODUCTION

In everyday life, we encounter numerous situations in which we have to direct attention selectively to a particular perceptual modality (e.g., visual, auditory, tactile) in order to acquire information necessary for achieving our current action goals. Whether we are looking for a book in the library, listening to a conversation at a cocktail party, or evaluating the surface texture of an object via tactile sensing, our brain employs some top–down perceptual set, or “template” of the objects of interest, to guide the extraction of the relevant information. Interestingly, the guidance becomes even more efficient when we attend to the same modality (e.g., touch; Spence, Nicholls, & Driver, 2001) or to the same dimension (e.g., color; Found & Müller, 1996) within one modality in successive perceptual episodes. That is, how efficiently we select relevant information is also determined by what (e.g., which modality) was selected just before.1

Shifting across Sensory Modalities

It is well established that focusing on the same perceptual modality in successive trial episodes (e.g., tactile target on both the current trial n and the preceding trial n − 1) facilitates performance, relative to when the modality changes across consecutive trials (e.g., tactile target on trial n preceded by visual target on trial n − 1). A large number of studies have used different experimental paradigms to investigate these modality repetition/change effects in normal subjects (e.g., Rodway, 2005; Gondan, Lange, Rösler, & Röder, 2004; Spence et al., 2001) as well as patients (e.g., Hanewinkel & Ferstl, 1996; Cohen & Rist, 1992; Manuzza, 1980; Verleger & Cohen, 1978; Levit, Sutton, & Zubin, 1973). For example, Rodway (2005) used a cueing paradigm to investigate the efficiency of warning signals. He found that, for brief foreperiods, the warning signal (cue) was most efficient when it was presented within the same, rather than a different, modality to the subsequent target. Rodway concluded that the warning signal exogenously attracts attention to its modality, thereby facilitating responses to subsequent targets defined within the same modality. A similar pattern was observed by Spence et al. (2001), who examined the effect of modality expectancy in a task that required participants to judge the azimuth (left vs. right) of the target location in an unpredictable sequence of auditory, visual, and tactile targets. There were two types of trial blocks: biased blocks in which the majority of targets (75%) was presented in one modality (participants were instructed to attend to this modality), and unbiased blocks in which the targets were equally likely to be defined in each modality (33%; participants were instructed to divide attention among the three modalities). With the majority of targets presented in one modality, Spence et al. observed prolonged RTs for targets defined within the unexpected compared to the expected modality. In trial blocks in which each target modality was equally likely, RT costs were observed for trials on which the modality changed relative to the preceding trial. In fact, such modality change costs were also evident in the biased trial blocks, accounting for almost all the benefits and for a large part of the costs in the “expectancy” relative to the “divided-attention” conditions.2 Spence et al. interpreted this pattern of results in terms of a stimulus-driven “modality shift effect” (MSE).

At the electrophysiological level, the effects accompanying modality changes have been linked to processes that operate in a modality-unspecific fashion, as well as to modality-specific processes within sensory brain areas. As indicated by several studies examining the performance difference between (schizophrenia) patients and normal controls, the MSE seems to modulate the amplitudes of the P3 component. However, the direction of this P3 amplitude effect varied across experimental studies. Whereas Verleger and Cohen (1978) and Levit et al. (1973) observed larger P3 amplitudes following modality changes relative to repetitions (in normal controls, but not in schizophrenics), a reversed effect has been reported by Rist and Cohen (1988). On the other hand, a recent study by Gondan et al. (2004) reported N1 amplitude modulations owing to modality shifts over modality-specific sensory areas. However, these MSE modulations varied depending on the respective modality: When the stimulus modality changed across trials, auditory N1 amplitudes were found to be enlarged while the amplitudes of the visual N1 component were decreased.

Although such modality repetition/change effects have been noted in the literature, there has been little systematic attempt to integrate these findings into a coherent theoretical framework. We propose that a model originally developed to account for dimension repetition/change effects within the visual (as well as the auditory) modality can be extended to account for the mechanisms underlying modality switch cost.

“Dimension Weighting” as a Model of “Modality Weighting”?

Similar to such modality change effects, sequential effects have also been reported in visual search for singleton feature targets, both when the target and distractor features were repeated or changed roles (e.g., Maljkovic & Nakayama, 1994) and when the target-defining dimension was repeated or changed across trials (e.g., Found & Müller, 1996; Müller, Heller, & Ziegler, 1995). In the latter case, the target could be defined by an odd-one-out feature within one of several possible dimensions (e.g., color, orientation), and participants were required to simply discern the presence (vs. the absence) of any target. Participants were faster to detect a target when the target-defining dimension remained the same on consecutive trials (e.g., a color-defined target on trial n following a color-defined target on trial n − 1), compared to when the target-defining dimension changed (e.g., a color-defined target on trial n following an orientation-defined target on trial n − 1). Importantly, this effect of dimension repetition was largely unaffected by changes of the target feature (e.g., red target on trial n, blue target on trial n − 1) within the repeated dimension (Found & Müller, 1996).3

To explain this set of findings, Müller and colleagues proposed a “dimension-weighting” account (DWA; e.g., Found & Müller, 1996; Müller et al., 1995). Similar to visual-search theories such as Guided Search (e.g., Wolfe, 1994), the DWA assumes that focal (selective) attention operates at a master map of integrated (summed) feature contrast signals derived separately in dimension-specific input modules. Detection of a singleton target requires that sufficient attentional weight is allocated to the corresponding dimension-specific input module, effectively amplifying its feature contrast signal and rendering it salient on the master map. The dimensional weight pattern established on a trial persists into the next trial, facilitating the processing of any subsequent target (whatever its feature description) defined within the same visual dimension. However, when the next target is defined in a different dimension, the wrong dimension is weighted initially, delaying target detection. In this case, a process is initiated in which attentional weight is shifted from the old to the new target-defining dimension—as a prerequisite for target detection and/or as a postselective adjustment process.

Recently, several studies have investigated the neural substrates of dimensional weighting using event-related potentials (ERPs; Töllner, Gramann, Müller, Kiss, & Eimer, 2008; Gramann, Töllner, Krummenacher, Müller, & Eimer, 2007) and event-related functional magnetic resonance imaging (fMRI; Pollmann, Weidner, Müller, & von Cramon, 2000, 2006; Pollmann, 2004; Weidner, Pollmann, Müller, & von Cramon, 2002). In the ERP study of Gramann et al. (2007), three components of the ERP were found to be associated with changes in the target-defining dimension on consecutive trials: Dimension changes were associated with an enhanced (anterior) transition N2 (tN2), delayed P3 latencies, and enhanced slow wave (SW) amplitudes. Gramann et al. interpreted the systematic modulation of the tN2 to reflect the detection of a dimension change and the initiation of the resetting of dimensional weights, whereas the P3 and SW were proposed to mediate the weight shifts via feedback pathways to dimension-specific input modules in higher-level visual areas. This pattern of ERP effects is in line with results from fMRI studies of Weidner et al. (2002) and Pollmann et al. (2000) identifying a fronto-posterior network to be sensitive to visual-dimension changes. Pollmann, Weidner, Müller, Maertens, and von Cramon (2006) and Pollmann, Weidner, Müller, and von Cramon (2006) concluded that prefrontal regions are the site of executive processes associated with the control of dimensional weight shifting (see also Pollmann et al., 2007), whereas higher visual areas in the superior parietal and temporal cortex mediate the weight shifts via feedback pathways to the dimension-specific input areas in the occipital cortex.

Rationale of the Present Study

By analyzing ERPs, the present study aimed at identifying electrocortical correlates that accompany modality switches independently of the current target modality, and thus, to provide further insights regarding the time course of behavioral MSEs. More specifically, it was examined whether an ERP component analogous to the tN2 component of the Gramann et al. (2007) study would be elicited as a consequence of modality changes across successive trial episodes. Recall that the tN2 component was previously found to be sensitive to visual-dimension changes, and thus, interpreted as reflecting a process of weight shifting that operates within sensory (e.g., visual, auditory) modalities. The presence of a similar ERP component that is sensitive to changes of the target modality might reflect a supramodal process that controls attentional weight shifting across sensory modalities (for previous research into supramodal attentional control processes in spatial attention, see Eimer & van Velzen, 2002; Farah, Wong, Monheit, & Morrow, 1989). This would have important implications with respect to the scope of the DWA. As noted above, Gramann et al. interpreted the tN2 to reflect the detection of a dimension change and the initiation of the resetting of dimensional (attentional) weights based on visual information. If the present study reveals an analogous component to reflect weight shifting across modalities, then a generalized “weighting account,” with an extended functional architecture, could be proposed to account for modality switching effects observed in earlier behavioral studies.

Taken together, the aim of the present study was (i) to confirm earlier findings of prolonged RTs for changes, relative to repetitions, of the target-defining modality and (ii) to identify an electrocortical correlate of this behavioral MSE that is elicited independently of the current target modality.

EXPERIMENT 1

Methods

Participants

Twelve paid volunteers (3 men, all right-handed, age range = 21–35 years, mean age = 27.9 years) recruited from the Birkbeck College subject panel gave their written informed consent to participate in the experiment. They all had normal or corrected-to-normal vision and reported having normal touch sensitivity. All were naïve as to the purpose of the study.

Stimuli and Apparatus

Participants were seated in a dimly lit and sound-attenuated experimental chamber. A 17-in. computer screen was placed centrally in front of the participant at a viewing distance of 55 cm. Tactile stimuli were presented using 5 mV solenoids, driving a metal rod with a blunt conical tip to the tip of the left and right index fingers. The index fingers were placed palm side down to the solenoids and were fixed using a Velcro strip. The rods made contact with the fingers whenever a current was passed through the solenoids. White noise was presented from a central loudspeaker (hidden behind the computer screen) throughout the experimental blocks to mask any sounds produced by the operation of the tactile stimulators. Visual stimuli were presented by illuminating a circular ensemble of seven green light-emitting diodes (LEDs; i.e., 6 LEDs arranged around 1 central LED). The angular size of each LED was 0.65°, and the circle diameter was 2.4° of visual angle. A white fixation cross against a black background was presented centrally at the bottom of the computer screen throughout the experimental blocks. Two tactile stimulators were positioned together with two visual stimulators 15 cm apart, 7.5 cm to either side to the fixation cross, and 50 cm from the edge of the table (from the participant's perspective) directly in front of the computer screen (see Figure 1). The LED ensembles were attached to the computer screen positioned 1 cm directly above the tactile stimulators. Tactile stimuli consisted of one rod contacting a finger for 200 msec, visual stimuli consisted of the illumination of one LED ensemble for 200 msec. To give a response, participants had to press either the left or the right foot pedal placed on the floor. The exact position of the foot pedals was adjusted for each participant individually to ensure a comfortable seating position.

Figure 1. 

Basic experimental set-up (top view). EEG was recorded while participants sat in front of a 17-in. monitor with index fingers placed palm side down to solenoids (embedded into foam; index fingers fixed using a Velcro strip). Single stimuli were presented either at the left or the right stimulus location (15 cm apart). Tactile stimuli consisted of a rod making contact with the tip of the index finger whenever a current was passed through the solenoids; visual stimuli were brief flashes of LEDs (placed directly above the solenoids). White noise was presented from a central loudspeaker (hidden behind the monitor), and participants had to press either a left or right foot pedal (placed on the floor) to give a response.

Figure 1. 

Basic experimental set-up (top view). EEG was recorded while participants sat in front of a 17-in. monitor with index fingers placed palm side down to solenoids (embedded into foam; index fingers fixed using a Velcro strip). Single stimuli were presented either at the left or the right stimulus location (15 cm apart). Tactile stimuli consisted of a rod making contact with the tip of the index finger whenever a current was passed through the solenoids; visual stimuli were brief flashes of LEDs (placed directly above the solenoids). White noise was presented from a central loudspeaker (hidden behind the monitor), and participants had to press either a left or right foot pedal (placed on the floor) to give a response.

Procedure

The experiment comprised 20 experimental blocks of 72 trials each. Trials started with the presentation of the fixation cross for 500 msec, followed by either a visual or a tactile stimulus for 200 msec. The trial was terminated by the participant's response or after a maximum duration of 1000 msec. The intertrial interval was 1000 ± 50 msec. On each trial, a single stimulus, either visual or tactile, was presented at one of the two possible locations. Participants were instructed to maintain eye fixation throughout the experimental block and to give a speeded forced-choice response indicating the modality of the stimulus. Half the participants responded with their left foot to visual stimuli and with their right foot to tactile stimuli, with the stimulus–response mapping changed after the first half of the experiment. For the other participants, the stimulus–response assignment was reversed. No feedback was given as to the correctness of the response. Visual and tactile stimuli were equally likely, and they were equally likely presented at the left and the right stimulus location. To further examine whether effects of modality changes might interact with the positional identity of the stimulus, all behavioral and electrophysiological data were analyzed with respect to the target modality and target position on the current trial n relative to preceding trial n − 1, resulting in four intertrial transition conditions: same modality–same position (sMsP), same modality–different position (sMdP), different modality–same position (dMsP), different modality–different position (dMdP). Prior to the start of each experimental half, participants performed at least one practice block.

Note that the presentation of only a single (lateral) stimulus in the present paradigm differs from previous studies investigating the DWA, which used visual-search tasks with a singleton target presented amongst a set of distracter stimuli. However, dimensional intertrial repetition/change effects are also found when the display contains only a single target defined in one of several visual dimensions (Mortier, Theeuwes, & Starreveld, 2005; see also Müller & O'Grady, 2000). Consequently, it was reasonable to expect modality repetition/change effects under the stimulus conditions employed in the present study.

EEG Recording and Data Analysis

The electroencephalogram (EEG) was recorded using Ag–AgCl electrodes mounted on an elastic cap (Falk Minow Service, Munich), referenced to linked earlobes. Electrode positions were a subset of the International 10–10 System sites (Fpz, F7, F3, Fz, F4, F8, FC5, FC6, T7, C3, Cz, C4, T8, CP5, CP6, P7, P3 Pz, P4, P8, PO7, PO3, PO4, PO8, O1, Oz, and O2). The horizontal electrooculogram (HEOG) was recorded from the outer canthi of both eyes. Data were recorded with BrainAmp amplifiers (Brain Products, Munich, Germany) using an analog bandpass from 0.1 to 40 Hz and a digitization rate of 500 Hz. All electrode impedances were kept below 5 kΩ.

Prior to epoching the EEG, an independent-component analysis, as implemented in the Brain Vision Analyzer (Brain Products) software, was performed to identify and eliminate blinks and horizontal eye movements. EEG data were epoched off-line into 1200-msec periods with a 200-msec prestimulus baseline. Note that only trials with correct responses on both the current and the preceding trial were selected for further analyses. The prestimulus period was used for baseline correction. Trials with signals exceeding ±60 μV were excluded from further analysis before the ERPs were averaged.

According to the DWA, processes associated with the control of (dimensional) attentional weighting are characterized as preattentive in locus (e.g., Töllner, Gramann, et al., 2008; Müller & Krummenacher, 2006). Therefore, we focused on early ERP components (P1, N1, N2) as potential markers for modality shifts irrespective of target modality. Mean amplitudes of these components were derived from visual inspection of the grand-average potentials (see Table 1) and examined using repeated measures ANOVAs, with the factors modality change (same vs. different modality), position change (same vs. different position), electrode site (frontal, central, parietal), and electrode position (left, midline, right), separately for each modality. These analyses were conducted for electrodes F3, Fz, F4, C3, Cz, C4, P3, Pz, and P4. Further analyses were conducted for early modality-specific ERP components (somatosensory P50 [45–75 msec] and N90 [85–115 msec] at electrodes C3/C4 contralateral to the stimulated hand; visual P1 [100–130 msec] and N1 [150–180 msec] at lateral occipital sites PO7/PO8) in order to investigate modality-specific modulations over early sensory areas that might additionally contribute to behavioral modality switch costs. Mean amplitudes of the early modality-specific ERP components were analyzed using repeated measures ANOVAs, with the factors modality change, position change, stimulus side (left vs. right), and electrode position (left vs. right). Because the experiment focused on the neural mechanisms underlying modality shifting, only main effects and interactions involving the factor “modality change” will be reported for the electrophysiological data. Whenever required, significant main effects and interactions were further examined using Tukey HSD post hoc contrasts.

Table 1. 

Time Windows for Calculating Mean Amplitudes for All Modality-unspecific ERP Components Examined in Experiment 1

Component
Mean Time Window (msec)
Recording Site (Left, Midline, Right)
Somatosensory P1 80–120 frontal, central, parietal 
Somatosensory N1 140–180 frontal, central, parietal 
Somatosensory N2 215–255 frontal, central, parietal 
Visual P1 70–110 frontal, central, parietal 
Visual N1 140–180 frontal, central, parietal 
Visual N2 230–270 frontal, central, parietal 
Component
Mean Time Window (msec)
Recording Site (Left, Midline, Right)
Somatosensory P1 80–120 frontal, central, parietal 
Somatosensory N1 140–180 frontal, central, parietal 
Somatosensory N2 215–255 frontal, central, parietal 
Visual P1 70–110 frontal, central, parietal 
Visual N1 140–180 frontal, central, parietal 
Visual N2 230–270 frontal, central, parietal 

Results

Behavioral Data

Trials on which participants responded incorrectly (4.93% of all trials), on which the RT was excessively slow (>1000 msec; 1.36% of all trials), and for which the response on the preceding trial was incorrect (4.35% of all trials) were excluded from further RT analysis (10.65% of all trials in total). Figure 2 displays the RTs and error rates for the remaining trials, for each of the four intertrial conditions. A repeated measures ANOVA of the RT data, with the factors modality (visual, tactile), modality change (same vs. different modality), and position change (same vs. different position), revealed a main effect of modality change [F(1, 11) = 30.33, p < .001, η2 = .734], with markedly slower reactions for modality changes compared to repetitions (511 vs. 461 msec). Furthermore, there was a main effect of position change [F(1, 11) = 10.48, p < .008, η2 = .488], with slower reactions for position changes relative to repetitions (490 vs. 481 msec). The Modality change × Position change interaction was also significant [F(1, 11) = 75.97, p < .001, η2 = .874]. This interaction was due to an increased RT advantage for modality repetition (as compared to change) trials when the target position was also repeated (as compared to changing); in contrast, with modality changes, RTs were faster when the position was also changed. Post hoc contrasts confirmed that RTs were significantly different between all four experimental conditions (p < .001). Response speed was marginally dependent on the sensory modality of the stimulus [visual vs. tactile: 479 vs. 492 msec; main effect of modality: F(1, 11) = 3.21, p > .101, η2 = .226].

Figure 2. 

RTs (lines) and error rates (bars) as a function of modality change and position change (sM = same modality; dM = different modality).

Figure 2. 

RTs (lines) and error rates (bars) as a function of modality change and position change (sM = same modality; dM = different modality).

An analogous ANOVA on the error rates revealed the main effects of modality [F(1, 11) = 11.12, p < .007, η2 = .503] and position change [F(1, 11) = 7.82, p < .017, η2 = .416] to be significant, with slightly fewer errors in response to visual as compared to tactile stimuli (4.2% vs. 5.6%) and for repetitions as compared changes of the stimulus position (4.5% vs. 5.4%). The interaction between modality change and position change was also significant [F(1, 11) = 18.57, p < .001, η2 = .658]. As can be seen from Figure 2, this interaction was due to fewer errors being made for modality repetition (compared to change) trials when the position was repeated, relative to being changed. The reversed pattern was observed for modality change trials. This pattern of effects indicates that RT effects were not confounded by speed–accuracy tradeoffs.

Effects on Somatosensory ERPs

ERPs elicited in response to somatosensory stimuli are presented in Figure 3A, separately for each of the four experimental conditions. No main effects of any of the experimental variables were obtained for the P1 amplitudes. Although there was a three-way interaction between modality change, position change, and electrode site [F(2, 22) = 3.85, p < .037, η2 = .259] for P1 amplitudes, this was not further substantiated by reliable main effects or interactions in follow-up analyses conducted separately for different electrode sites.

Figure 3A. 

Grand-averaged ERP waveforms elicited in response to somatosensory stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate position repetitions, dark gray lines position changes.

Figure 3A. 

Grand-averaged ERP waveforms elicited in response to somatosensory stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate position repetitions, dark gray lines position changes.

As can be seen from Figure 3A, modality changes were associated with enhanced amplitudes of the N1 component in the 140–180 msec time window,4 validated by a significant main effect of modality change [F(1, 11) = 10.82, p < .007, η2 = .496]. There was no significant main effect of position change [F(1, 11) = 0.45] and no Modality change × Position change interaction [F(1, 11) = 1.49], demonstrating that this N1 modulation was solely linked to changes versus repetitions of the target modality.

No effects involving modality change were observed for N2 amplitudes.

Figure 3B shows somatosensory ERPs as a function of Modality change × Position change at electrodes C3/C4. As expected, the early somatosensory P50 and N90 components were only elicited contralaterally to the stimulated hand.

Figure 3B. 

Grand-averaged ERP waveforms elicited over modality-specific sensory areas at electrode positions C3/C4 by tactile stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate position repetitions, dark gray lines position changes.

Figure 3B. 

Grand-averaged ERP waveforms elicited over modality-specific sensory areas at electrode positions C3/C4 by tactile stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate position repetitions, dark gray lines position changes.

Although there was no significant effect of modality change on P50 amplitudes, the subsequent N90 was enhanced for modality change trials, substantiated by a significant main effect of modality change [F(1, 11) = 9.57, p < .010, η2 = .465]. Again, there was no interaction between modality change and position change [F(1, 11) = 1.22], demonstrating that this early effect of modality change is independent of changes versus repetitions of stimulus locations (see also Figure 3A).

Effects on Visual ERPs

Figure 4A displays ERPs elicited in response to visual stimuli, separately for each of the four experimental conditions. No significant effects or interactions involving the factor modality change were found for the visual P1 component.

Figure 4A. 

Grand-averaged ERP waveforms elicited in response to visual stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate position repetitions, dark gray lines position changes.

Figure 4A. 

Grand-averaged ERP waveforms elicited in response to visual stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate position repetitions, dark gray lines position changes.

In contrast, and analogous to the results found for somatosensory ERPs, the N1 component was strongly affected by modality change, with significantly larger N1 amplitudes for trials on which the target modality was changed [main effect of modality change, F(1, 11) = 7.94, p < .017, η2 = .419]. As was already observed for tactile ERPs, no significant main effect of position change [F(1, 11) = 0.079] and no Modality change × Position change interaction [F(1, 11) = 1.56] were obtained for visual N1 amplitudes—thus confirming that N1 amplitude modulations were associated with modality changes versus repetitions, irrespective of whether successive stimuli were presented at matching locations or in opposite hemifields.

For visual N2 amplitudes, the interaction between modality change, electrode site, and electrode position reached significance [F(1, 11) = 3.73, p < .011, η2 = .253]. However, this was not substantiated by significant main effects or interactions in follow-up analyses conducted separately for different electrode sites.

Figure 4B presents the early sensory evoked potentials specific for the vision modality over early visual areas at electrode positions PO7/PO8, separately for each of the four experimental conditions.

Figure 4B. 

Grand-averaged ERP waveforms elicited over modality-specific sensory areas at electrode positions PO7/PO8 by visual stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate position repetitions, dark gray lines position changes.

Figure 4B. 

Grand-averaged ERP waveforms elicited over modality-specific sensory areas at electrode positions PO7/PO8 by visual stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate position repetitions, dark gray lines position changes.

Statistical analyses revealed that visual P1 and N1 components were both affected by shifts of the stimulus-defining modality across consecutive trials. Sensory evoked P1 amplitudes were modulated by modality changes interacting with stimulus side [F(1, 11) = 6.96, p < .023, η2 = .388]. This interaction was based on significantly enhanced P1 amplitudes following modality changes if the stimulus appeared within the right (p < .015), but not the left (p < .403), hemifield.

Sensory evoked N1 amplitudes were modulated by modality changes interacting with stimulus side and electrode position [F(1, 11) = 4.94, p < .048, η2 = .310]. This three-way interaction was due to MSEs observable at ipsilateral, but not contralateral, electrode positions, that is, left hemifield stimuli produced enhanced amplitudes owing to modality shifts at PO7 (p < .043), but not PO8 (p > .455); conversely, right hemifield stimuli generated increased activations owing to modality shifts at PO8 (p < .011), but not PO7 (p < .109).

Comparison of N1 Modality Shift Effect across Modalities

Further analyses were conducted to verify whether the N1 modulation produced by a change in target modality across successive trials, which was observed for both visual and somatosensory ERPs within the same time range, represents a modality-unspecific process, or, alternatively, a process operating in a modality-specific fashion. This was examined by subjecting N1 mean amplitude values for both stimulus modalities to an omnibus ANOVA, with the additional factor modality (touch, vision). As expected, this ANOVA revealed significant main effects of modality [F(1, 11) = 37.08, p < .001, η2 = .771] and modality change [F(1, 11) = 87.81, p < .001, η2 = .889] as well as an interaction between Modality × Electrode site [F(1, 11) = 21.31, p < .001, η2 = .660]. In contrast, and importantly, the interaction between modality and modality change was far from significant [F(1, 11) = 1.08, p > .320, η2 = .090], indicating that the N1 amplitude modulations resulting from modality changes were triggered in an equivalent fashion regardless of whether visual or tactile target stimuli were presented.

Discussion

As expected, the RT data confirmed previous findings (e.g., Spence et al., 2001) of faster reactions when the current target was defined within the same, rather than a different, modality relative to the preceding target. However, performance was also determined by the position of the stimulus. RTs were fastest when both the modality and the position of the target were repeated and slowest when the target modality was changed but the position was repeated, with intermediate response latencies in the two remaining conditions. Thus, concurrent changes of modality and position did not produce additive effects, indicating that an interaction of modality-related and positional information processing must occur at some stage of processing. This interactive behavior is in accordance with previous studies (Roeber, Berti, Widmann, & Schröger, 2005; Kleinsorge, 1999), which revealed a bias for changing the response when a response-irrelevant feature had been changed (rather than repeated), and has been linked to processes involved in “response selection” (see Töllner, Gramann, et al., 2008). However, because modality changes were associated with response changes in Experiment 1, it is not unequivocally clear at which stage of processing, perceptual versus response-related, this modality-specific intertrial facilitation arises.

At the electrophysiological level, modality changes affected the N1 component, independently of the target modality. For both somatosensory and visual stimuli, changes of the target modality on consecutive trials were associated with enhanced N1 amplitudes, relative to modality repetitions. Importantly, the modulation of the N1 was independent of the perceptual modality and repetitions/changes of the stimulus position, suggesting that the N1 effect originates from a purely “modality change-driven” process. According to a generalized weighting account (along the lines of the DWA; Found & Müller, 1996), the enhanced amplitudes of the N1 component in response to modality changes might be interpreted as reflecting a control mechanism which is invoked to detect a (modality) change necessary to transfer attentional weight from the old to the new target-defining modality. Thus, optimized stimulus processing in the subsequent trial episode is accomplished by rendering the new target signal (more) salient at some supramodal decision stage (see General Discussion for a more detailed discussion).

This hypothesized processing architecture is further supported by the results observed for the early sensory evoked potentials specific for somatosensory (N90) and visual (visual P1 and N1) processing, which suggest that shifts of the target modality across consecutive trials led to differences already in the early sensory stages of information processing, possibly coding modality-specific information with differential efficiency. Importantly, there were no main effects of position change, or interactions between modality change and position change, demonstrating that the amplitudes of these components (as well as the amplitudes of the modality-unspecific N1) were not affected by possible sensory refractoriness effects that might have been present when two tactile or two visual stimuli were presented on successive trials at identical locations (see also Discussion of Experiment 2).

EXPERIMENT 2

Experiment 2 was designed to rule out the possibility that the modulation of the N1 component as a result of modality changes versus repetitions observed in Experiment 1 was attributable to repetitions/changes of the motor response. Because a modality change was invariably associated with a response change in Experiment 1, it is not possible to decide whether the modality change effects are attributable to perceptually related processes, response-related processes, or an interaction of both. To address this question, we introduced two features per modality in Experiment 2, with one feature in each modality mapped to the same motor response (e.g., “green” and “slow vibrating” → left foot; “red” and “fast vibrating” → right foot). Using this stimulus–response mapping, a modality change could occur independently of repetitions/changes in the motor response.

Methods

Participants

Twelve paid volunteers (3 men, all right-handed, age range = 21–35 years, mean age = 27.3 years) were recruited from the Birkbeck College subject panel, after giving their written informed consent. One participant had to be excluded from data analysis due to excessive eye-blink artifacts.

Stimuli, Apparatus, and Procedure

The general experimental set-up and procedure were the same as in Experiment 1, except for the introduction of two features for each modality. Tactile stimuli were vibrations that differed in frequency. To present “slow” vibrations, the contact time of the rod to the finger was set to 2 msec, followed by a 23-msec interpulse interval. This corresponded to a rectangular stimulation frequency of 40 Hz. “Fast” vibrations were defined by a contact time of 2 msec and an interpulse interval of 8 msec, corresponding to a rectangular stimulation frequency of 100 Hz. These manipulations of the contact times and interpulse intervals resulted in two easily discriminable vibratory stimuli (40 Hz vs. 100 Hz). The duration of the stimuli (the interval between onset of the first pulse and the offset of the last pulse) was set to 200 msec. Visual stimuli consisted of illuminating an LED ensemble for 200 msec, as in Experiment 1. However, LEDs now differed in color (red or green). Prior to each experimental half, participants were informed about the required stimulus–response mapping. Fifty percent of the participants responded with their left foot to red and slow vibrating stimuli, and with their right food to green and fast vibrating stimuli, in the first half of the experiment, and vice versa in the second half. This was reversed for the other participants. Prior to the start of each experimental half, participants performed at least one trial block to practice the stimulus–response mapping. The defining features (red, green, slow vibrating, fast vibrating) and positions (left, right) of the target stimuli, as well as the required motor responses, were equally likely (and presented in random order across trials). All behavioral and electrophysiological data were analyzed with respect to the target modality, target position, and motor response on the current trial n relative to preceding trial n − 1, thus adding to the four experimental conditions of Experiment 1 the factor response change (same vs. different response), which resulted in eight intertrial transition conditions (all with equal numbers of trials).

Statistical analyses of the electrophysiological data were focused primarily on the N1 component, which was found to be a modality-independent electrocortical marker of modality shifting in Experiment 1. Mean amplitudes (identical time range as in Experiment 1) of the N1 were examined using a repeated measures ANOVA with the factors modality change (same modality, different modality), response change (same response, different response), position change (same position, different position), electrode site (frontal, central, parietal), and electrode position (left, midline, right), separately for each modality. Additionally, mean amplitudes of the early somatosensory contralateral P50 and N90 components were subjected to repeated measure ANOVAs with the factors modality change (same vs. different modality), response change (same vs. different response), position change (same vs. different position), and stimulus side (left vs. right) at C3/C4. An ANOVA with the factors modality change, response change, position change, stimulus side, and electrode position (left vs. right) was conducted to explore any effects on visual evoked P1 and N1 components at PO7/PO8. In all other respects (procedure, EEG recording, and data analysis), Experiment 2 was identical to Experiment 1.

Results

Behavioral Data

Trials on which participants responded incorrectly (5.53% of all trials), on which the RT was excessively slow (>1000 msec; 1.37%), and with an incorrect response on the previous trial (5.06% of all trials) were excluded from further RT analyses (11.96% of the trials in total). RTs and error rates for the remaining trials are displayed as a function of Modality change × Response change in Figure 5. A repeated measures ANOVA of the RT data, with the factors modality (visual, tactile), modality change (same vs. different modality), response change (same vs. different response), and position change (same vs. different position) revealed significant main effects for modality, modality change, and response change. The modality effect [F(1, 10) = 27.61, p < .001, η2 = .734] was caused by faster reactions for visual compared to tactile targets (546 vs. 595 msec). The modality change effect [F(1, 10) = 67.99, p < .001, η2 = .872] was due to slowed responses for modality changes relative to repetitions (596 vs. 545 msec). The response change effect [F(1, 10) = 33.82, p < .001, η2 = .772] was due to prolonged RTs for response changes compared to repetitions (584 vs. 557 msec). In addition, the interaction between modality change and response change was significant [F(1, 10) = 20.20, p < .001, η2 = .669].

Figure 5. 

RTs (lines) and error rates (bars) as a function of modality change and response change (sM = same modality; dM = different modality).

Figure 5. 

RTs (lines) and error rates (bars) as a function of modality change and response change (sM = same modality; dM = different modality).

Further analyses confirmed that participants reacted fastest when both the modality and the response stayed the same on consecutive trials, followed by trials on which the modality was repeated and the response changed (p < .005). With modality changes, RTs did not differ between trials on which the response was repeated versus changed (p > .967). The factor response change interacted further with position change [F(1, 10) = 8.44, p < .016, η2 = .458]: A change of the required motor response resulted in slower RTs for position repetition than for position change trials. This observation was confirmed by further analyses. For position repetition trials, RTs were significantly slower for response changes relative to response repetitions (p < .001). For position change trials, the difference between same and different responses failed to reach significance (p > .06). Finally, the three-way interaction between modality, modality change, and position change was significant [F(1, 10) = 7.36, p < .022, η2 = .424]. As revealed by further post hoc contrasts, responses on tactile–modality repetition trials were faster when the target appeared at the same position as on the previous trial (p < .001). In contrast, there was no such influence of position repetitions/changes on visual modality repetition trials (p > .727).

An analogous ANOVA of the error rates revealed that participants made significantly fewer errors on modality repetition compared to change trials (3.6% vs. 7.5%) [main effect of modality change: F(1, 10) = 14.53, p < .003, η2 = .592]. This indicates that the RT effects in Experiment 2 were not confounded by speed–accuracy tradeoffs.

Effects on Somatosensory ERPs

Similar to Experiment 1, the main effect of modality change was significant for the somatosensory N1 amplitudes [F(1, 10) = 6.46, p < .029, η2 = .393]. As can be seen from Figure 6A, N1 amplitudes were enhanced for modality changes versus repetitions. There was no significant main effect of response change [F(1, 10) = 0.52], and no Modality change × Response change interaction [F(1, 10) = 1.86], demonstrating that this N1 modulation was solely linked to changes versus repetitions of the target modality. Figure 6B shows somatosensory ERPs on modality change and modality repetition trials at electrodes C3/C4 contralateral to the stimulated hand.

Figure 6A. 

Grand-averaged ERP waveforms elicited in response to somatosensory stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate response repetitions, dark gray lines response changes.

Figure 6A. 

Grand-averaged ERP waveforms elicited in response to somatosensory stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate response repetitions, dark gray lines response changes.

Figure 6B. 

Grand-averaged ERP waveforms elicited over modality-specific sensory areas at electrode positions C3/C4 by tactile stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate response repetitions, dark gray lines response changes.

Figure 6B. 

Grand-averaged ERP waveforms elicited over modality-specific sensory areas at electrode positions C3/C4 by tactile stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate response repetitions, dark gray lines response changes.

As for Experiment 1, amplitude modulations due to modality changes were evident for the N90, but not for the P50 component. For the N90 amplitudes, a significant main effect of modality change [F(1, 10) = 6.13, p < .033, η2 = .380] was found, due to enhanced amplitudes on modality change trials. In addition, and in contrast to the results found for Experiment 1, there was now also an interaction between modality change and position change [F(1, 10) = 7.74, p < .019, η2 = .436]. This interaction was due to significantly enhanced amplitudes following modality shifts occurring at the same location (p < .008), but not at the opposite location (p > .989), relative to the previous trial.

Effects on Visual ERPs

As can be seen from Figure 7A, changes of the target-defining modality were associated with more negative-going deflections of the N1 component, as compared to modality repetitions [main effect of modality change: F(1, 10) = 5.87, p < .036, η2 = .370]. In addition, there was an (marginally significant) interaction between modality change, electrode site, and electrode position revealed [F(4, 40) = 2.51, p < .057, η2 = .201]. This interaction reflects the fact that enhanced negativities owing to modality changes were most pronounced at frontal electrode positions, whereas this effect decreased toward midline and right central electrode positions, and was almost absent at midline and right parietal electrode positions. As with the tactile N1 amplitudes, there was no significant main effect of response change [F(1, 10) = 0.44], and no Modality change × Response change interaction [F(1, 10) = 0.02] on visual N1 amplitudes, assuring that this N1 modulation was not affected by changes versus repetitions of the motor response.

Figure 7A. 

Grand-averaged ERP waveforms elicited in response to visual stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate response repetitions, dark gray lines response changes.

Figure 7A. 

Grand-averaged ERP waveforms elicited in response to visual stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate response repetitions, dark gray lines response changes.

Figure 7B presents the early sensory evoked potentials specific for the visual modality, as a function of Modality change × Response change. Similar to Experiment 1, the early visual evoked P1 and N1 were influenced by the defining modality of the preceding stimulus. However, this time, the modality change factor interacted with electrode position [P1: F(1, 10) = 7.89, p < .019, η2 = .441; N1: F(1, 10) = 8.98, p < .013, η2 = .473]. For both components, shifts of the stimulus-defining modality were accompanied by unilateral amplitude enhancement at either the left (N1) or the right (P1) electrode position.

Figure 7B. 

Grand-averaged ERP waveforms elicited over modality-specific sensory areas at electrode positions PO7/PO8 by visual stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate response repetitions, dark gray lines response changes.

Figure 7B. 

Grand-averaged ERP waveforms elicited over modality-specific sensory areas at electrode positions PO7/PO8 by visual stimuli in the 300-msec interval following stimulus onset, relative to a 200-msec prestimulus baseline. Solid lines indicate modality repetitions, dotted lines modality changes. Light gray lines indicate response repetitions, dark gray lines response changes.

Comparison of N1 Modality Shift Effect across Modalities

As for Experiment 1, N1 mean amplitude values for both modalities were subjected to an omnibus ANOVA in order to investigate the modality independence of the N1 MSE. The results exactly replicated the pattern observed for Experiment 1. There were main effects of modality [F(1, 10) = 25.47, p < .001, η2 = .718] and modality change [F(1, 10) = 13.25, p < .005, η2 = .570], as well as an interaction between modality and electrode site [F(1, 10) = 21.28, p < .001, η2 = .680]. In contrast, there was no sign of differential activation patterns between trials with tactile and trials with visual stimuli, evidenced by the absence of a significant Modality × Modality change interaction [F(1, 10) = 1.00, p < .341, η2 = .091]—in line with the assumption that the enhanced N1 amplitude following a change, versus a repetition, of the target modality is modality-unspecific in nature.

Discussion

The aim of Experiment 2 was to confirm the results of Experiment 1, while at the same time ruling out potential contributions of response repetitions versus alternations. This was done by using a stimulus–response mapping that allowed modality changes to occur independently of response changes and vice versa. The RT data of Experiment 2 suggest an interactive behavior of the two factors. Repetitions/changes of the motor response influenced performance on modality repetition trials, with faster RTs when the response was repeated as well as the modality. However, no such influence was evident for modality change trials, on which RTs were generally slower compared to modality repetition trials. This interactive pattern of effects resembles that observed in previous studies (e.g., Töllner, Gramann, et al., 2008; Müller & Krummenacher, 2006; Pollmann, Weidner, Müller, Maertens, et al., 2006), which used a “compound task” to dissociate perceptually related from response-related processes in cross-dimensional singleton feature search. In these studies, participants produced the fastest responses when both the target-defining dimension and the response remained the same across consecutive trials. Changes of the visual dimension, the response, or both, all slowed the RTs to a similar level. Based on their ERP findings, Töllner et al. proposed that the interaction between the two factors arises at the “response selection” stage where perceptually analyzed information is translated into motor commands.

Confirming the observations of Experiment 1, the N1 component was modulated by modality changes in the same manner for somatosensory and for visual stimuli. Changes of the modality (from somatosensory to visual and vice versa) across consecutive trials were, irrespective of the perceptual modality and stimulus position (same vs. different as on the previous trial), associated with significantly enhanced N1 amplitudes. Importantly, this N1 effect occurred independently of repetitions/changes of the motor response, thereby ruling out any contribution of response-related factors. In Experiment 2, the visual MSE of the N1 component was most pronounced at frontal leads and almost disappearing toward central and parietal leads, revealing a fronto-central process involved in modality shifting. This observation resembles the findings of Gramann et al. (2007), suggesting that analogous brain regions are involved in the N1 MSE observed in the present study as in the tN2 visual-dimension shift effect described by Gramann and colleagues.

It should be noted that, in theory, there might have been sensory refractoriness effects involved in the elicitation of the (anterior) N1 component that operate in a modality-specific, but location-unspecific fashion. However, although it might be possible that processes localized in early sensory areas contribute to a fronto-central ERP component, the possibility that sensory refractoriness effects can account for the modality-unspecific N1 activation pattern observed in the present study is highly unlikely for several reasons. First, the primary determination of a component's decrement due to refractory processes is the time interval between stimuli. This influence of the interstimulus interval (ISI) on refractory processes was systematically studied in recovery-cycle studies (for a review, see Loveless, 1983) demonstrating dramatic increases in the N1 component from 0.5 to 2 sec and only gradually increases thereafter.5 In the present study, ISIs were 1.5 sec plus mean RT of around 530 msec, summing up to 2.03 sec, on average, between two successive stimulus presentations. Thus, the ISI chosen in the present study should be sufficiently long to exclude dramatic refractoriness effects. Further, to our knowledge, recovery cycle effects are most pronounced for identical features but not distinct features within the same modality. Experiment 2 of the present study clearly demonstrated that a change of features did not reveal any recovery cycle effects. Finally, it seems rather unlikely that neurons concerned with distinct (visual and tactile) sensory information processing show exactly the same temporal characteristics of sensory refractoriness. In the unlikely case of identical temporal characteristics, cortico-cortical connections of the visual and somatosensory cortices to fronto-central areas are highly dissimilar, rendering it unlikely that both modality-specific (tactile and visual) activations arrive/accumulate at exactly the identical time point at frontal regions to affect the activation pattern of the anterior N1 component.

Taken together, it appears implausible that sensory refractory processes—that result from a repetitions of different features and originate in two distinct sensory modalities at spatially distinct brain areas—would lead to the exact same modality-unspecific N1 decrement over fronto-central electrode positions as observed in the present study (which is further confirmed by the absence of a significant Modality × Modality change interaction).

Mirroring Experiment 1, modulated processing owing to modality shifts was also obtained for the early sensory evoked components. Albeit interacting with other factors (tactile N90: position change;6 visual P1 and N1: electrode position), the results clearly demonstrated an influence of the previous perceptual modality on early tactile and visual processing. As for Experiment 1, these modulations might indicate differences in processing efficiency starting already in the modality-specific sensory brain regions.

GENERAL DISCUSSION

In two ERP experiments investigating modality switch costs between vision and touch, we replicated the RT pattern described in previous studies (e.g., Spence et al., 2001): Changes in the target-defining modality across consecutive trials gave rise to prolonged RTs, compared to repetitions of the target modality. The purpose of the present study was to identify EEG parameters associated with this modality switch cost. A recent study of dimension change effects in the visual modality (Gramann et al., 2007) had revealed the tN2 component as a marker of visual-dimension changes. This effect was strongest over fronto-central electrode positions, pointing to the involvement of a frontal executive process in the control of visual-dimension (re-)weighting. The present study was modeled after this earlier study, and examined whether visual dimension changes (as studied by Gramann et al., 2007) and modality changes may be controlled by similar processes originating from similar brain regions. Specifically, a fronto-central ERP component analogous to the tN2 was expected to be sensitive to modality changes.

Brain Electrical Activity of Modality Changes

Analyses of ERPs revealed enhanced amplitudes of the N1 component for changes, relative to repetitions, of the target-defining modality. Importantly, the N1 MSE was observed in response to both visual and tactile target changes in Experiment 1, suggesting a process that operates independently of and across sensory modalities. To examine whether the N1 component reflects change processes originating from perceptual versus response-related processing stages, Experiment 2 was conducted with modality changes occurring independently of response changes. Similar to Experiment 1, the N1 exhibited enhanced amplitudes for modality changes relative to repetitions, irrespective of the perceptual modality, spatial stimulus characteristics, and motor response requirements. This pattern strongly suggests that the N1 effect reflects a mechanism based solely on nonspatial perceptual stimulus attributes—consistent with theoretical accounts (such as DWA) that locate intertrial change/repetition effects at perceptual processing stages, and inconsistent with accounts that attribute such effects exclusively to response-related stages (e.g., Mortier et al., 2005).

In Experiment 2, the N1 MSE was most pronounced in response to visual stimuli at frontal leads, with a significant decrease toward parietal leads. This finding is of special relevance with respect to the primary aim of the present study, namely, to identify an ERP marker mirroring modality shifts irrespective of the perceptual modality. Note that only the anterior portion (possibly originating from fronto-centrally located sources) of the N1 component exhibited this characteristic behavior for both modalities in both experiments. This accentuation of fronto-central electrode sites for the N1 MSE revealed an analogous scalp distribution to that observed by Gramann et al. (2007) for the tN2 in response to visual dimension changes. It is therefore possible that the anterior N1 MSE observed in the present study and the tN2 reported by Gramann et al. originate from similar brain regions, despite the fact that their latency differed by about 100 msec. This latency difference might be due to the absence of a time-demanding search process in the present study. In the study of Gramann et al., participants had to search for a color- or orientation-defined singleton target among distracters. In contrast, in the present study, participants were always presented with a single stimulus, either visual or tactile, so that there was no need for a search process prior to target discrimination. Admittedly, the assumption of an identical neural generator for the anterior N1 and the tN2 remains speculative and will require additional source reconstruction based on high-density EEG recording. Nonetheless, given its fronto-central focus, latency, and modulation independent of the target modality, stimulus location, and motor requirements, we interpret the anterior N1 as being associated with the control of modality-specific attentional weighting, that is, the detection of a modality change and initiation of the resetting of weights to the new target-defining modality.

Thus, put into a broader (ERP) context, the present findings suggest that shifting between perceptual modalities should be added to the kinds of processes (e.g., Vogel & Luck, 2000: discrimination; Näätänen, Jacobsen, & Winkler, 2005: sensory memory; Gehring, Goss, Coles, Meyer, & Donchin, 1993: error detection) that modulate, and are accomplished within the time range of, the N1 component. Critically, in close resemblance to the tN2 dimension change effect (see also N270; Wang, Cui, Wang, Tian, & Zhang, 2004), the N1 MSE is defined by enhanced amplitudes accompanying prolonged RTs, whereas the reverse pattern has been reported for other fronto-central ERP components (e.g., enhanced mismatch negativity amplitudes associated with faster RTs and increased hit rates; see Tiitinen, May, Reinikainen, & Näätänen, 1994). Accordingly, this characteristic activation pattern underscores our notion of a process engaged in attentional weight shifting, as opposed to preattentive sensory memory processes (as assumed for the mismatch negativity) underlying the present N1 (modality shift) effect.

In agreement with our weighting approach, and with the study of Gondan et al. (2004), are the results for the early sensory evoked potentials obtained in the present study. In both experiments, early sensory modality-specific components were affected by shifts of the stimulus-defining modality across consecutive trials. This suggests that already early sensory stages of information processing are modulated by modality shifts, and thus, might be contributing to behavioral modality switch costs. These modulations over modality-specific brain areas can be interpreted as reflecting the (implicit) weighting of one sensory stimulus modality over others, initiated via feedback pathways by frontal control mechanisms.

Introducing a “Modality-weighting” Account

The present findings revealed remarkable similarities between visual-dimension changes (Gramann et al., 2007) and modality changes (present study). In both studies, the behavioral RTs were prolonged for changes, relative to repetitions, of the target-defining visual dimension and modality, respectively. Furthermore, the electrophysiological data suggest spatially overlapping neural sources contributing to both types of change effect. On this basis, we propose a “modality-weighting” account (MWA), which is essentially a generalization of the DWA. Specifically, the MWA assumes similar weighting mechanisms for perceptual modalities as assumed for dimensions within the visual (and the auditory, e.g., Dyson & Quinlan, 2002) modality. That is, to optimize task performance, attentional processing weight is allocated to task-relevant stimulus modalities (such as vision, audition, touch), with the total weight being limited. Weighting of one modality leads to facilitated processing of all targets defined in this modality, relative to targets defined in other modalities. This facilitation results from enhanced coding of target signals within the weighted modality and/or enhanced transmission of modality-specific target information to a cross-modal stage of processing (such as a supramodal master map of locations), which determines the allocation of focal (selective) attention to the target event and mediates further perceptual analysis and response decisions (Figure 8).

Figure 8. 

Functional architecture of the “modality-weighting” account (MWA), adapted from the DWA (e.g., Found & Müller, 1996), with additional modality (saliency) maps placed between intramodal dimension maps and the supramodal master map unit. The example illustrated is a singleton feature search trial, with a singleton defined within the visual dimension “color.” The allocation of selective (focal) attention is determined by the distribution of activity on the master map, the units of which integrate (sum) saliency signals from separate modality-specific maps, which in turn receive signals from separate dimension-specific input modules. It is assumed that signal transmission between dimension- and modality-specific maps and between the latter and the master map is weighted depending on the target on the previous trial. The situation depicted shows essentially a bottom–up search. However, the MWA assumes interacting bottom–up and top–down mechanisms contributing to target detection (cf. Töllner, Zehetleitner, Gramann, & Müller, submitted; Müller et al., 2003).

Figure 8. 

Functional architecture of the “modality-weighting” account (MWA), adapted from the DWA (e.g., Found & Müller, 1996), with additional modality (saliency) maps placed between intramodal dimension maps and the supramodal master map unit. The example illustrated is a singleton feature search trial, with a singleton defined within the visual dimension “color.” The allocation of selective (focal) attention is determined by the distribution of activity on the master map, the units of which integrate (sum) saliency signals from separate modality-specific maps, which in turn receive signals from separate dimension-specific input modules. It is assumed that signal transmission between dimension- and modality-specific maps and between the latter and the master map is weighted depending on the target on the previous trial. The situation depicted shows essentially a bottom–up search. However, the MWA assumes interacting bottom–up and top–down mechanisms contributing to target detection (cf. Töllner, Zehetleitner, Gramann, & Müller, submitted; Müller et al., 2003).

In contrast, changes of the target-defining modality across consecutive trials involve a time-consuming weight-shifting process, in which attentional weight is transferred from the old to the new target-defining modality to amplify the target signal and render it salient at a supramodal processing stage (master map). The modulation of the anterior N1 component observed in the present study is assumed to reflect this weight-shifting process across modalities. Thus, regarding the time course of the processes involved in (implicit) attentional weight-setting, it is suggested that the anterior N1 MSE is primarily generated on the current trial, keeping track of the prevailing stimulus modality in order to adjust/update the weight-setting for optimized stimulus processing in the next trial episode. By contrast, early sensory-specific ERP effects of modality repetitions represent the facilitated sensory coding of the relevant stimulus modality as a consequence of the previous trial.

It is important to note that, according to this account (Figure 8) as well as other saliency-based models of attentional processing (e.g., Guided Search, Wolfe, 1994; DWA, Müller et al., 1995), a master map unit only signals, or knows, that there is a signal difference at one location relative to others, but not what precise feature contrast this difference is based on. Thus, target detection can be accomplished even without explicit knowledge about the target's featural or dimensional identity. However, if the target's identity is needed to accomplish a task (e.g., to map a specific sensory feature to a specific motor response, as in Experiment 2), recurrent processes have to feed back from the master map to hierarchically lower stages (e.g., to the stage of modality maps in Experiment 1, or the stage of feature maps in Experiment 2) in order to reveal the information of interest.7 Note that this gradual back-tracking architecture advocated here resembles the feedback mechanisms proposed in the Reverse Hierarchy Theory (RHT; Ahissar & Hochstein, 2004; Hochstein & Ahissar, 2002), originally developed to explain perceptual learning. According to RHT, the extraction of detailed (object) information depends on the operation of feedback connections from high-level to low-level processing areas, until a sufficient signal-to-noise ratio is available (see also Lamme & Roelfsema, 2000 for a detailed review of feedforward and recurrent connections within the visual modality).

Finally, it should be noted that modality weighting is theoretically consistent with (intramodality) dimension weighting: The weighting mechanisms for modalities and (intramodality) dimensions may be operating in tandem, modulating simultaneously the emergence of an overall saliency signal at the level of the supramodal master map. However, it remains an open issue whether modality weighting and dimension weighting involve one-and-the-same limited-capacity resource, or whether each modality has its “own” resource limitation, which determines the distribution of dimensional weights within the respective modality. This is equivalent to the question whether switching between dimensions of different modalities occurs at the same level as switching between dimensions within one modality, or whether shifting between modalities occurs at a higher level, as assumed in the MWA framework depicted in Figure 8.8 In summary, although the present data are consistent with a hierarchical MWA architecture, they do not provide unequivocal evidence in favor of a separate modality-specific selection level—so that the precise relationship between modality- and dimension-specific weighting mechanisms needs to be worked out in future studies. For these, based on the present findings of modality-specific weighting effects as well as previous findings of dimension-specific effects (e.g., Töllner, Gramann, et al., 2008; Müller et al., 1995), we hypothesize that optimized intertrial facilitation for a given target depends on (at least) two factors: first, as a precondition, the target modality must stay the same; and second, the dimension must be repeated.

Acknowledgments

This research was supported by German Research Foundation (DFG) grants FOR480 and EC142 (Excellence Cluster “CoTeSys”), and by a grant from the Biotechnology and Biological Sciences Research Council (BBSRC), UK, to M. E.

Reprint requests should be sent to Thomas Töllner, Department of Psychology, General and Experimental Psychology, Ludwig-Maximilians-University Munich, Leopoldstr. 13, D-80802 Munich, Germany, or via e-mail: toellner@psy.uni-muenchen.de.

Notes

1. 

As Maljkovic and Nakayama (1994) have demonstrated, this influence is strongest immediately after a given trial and decreases gradually over the following five to eight trials.

2. 

This pattern is similar to the dimension cueing effects revealed for the visual modality (see Müller, Reimann, & Krummenacher, 2003; Müller et al., 1995).

3. 

Similar effects have also been described for discriminations of the visual target dimension (e.g., color vs. orientation; Found & Müller, 1996) as well as for the auditory modality (e.g., Dyson & Quinlan, 2002).

4. 

This component is often also referred to as N140 in the somatosensory ERP literature. We describe this component here as N1 in order to highlight the similarities of ERP modality shift effects across touch and vision.

5. 

Nelson and Lassman (1968) reported increased amplitudes for ISIs ranging from 0.5 up to 10 sec; however, this (long-lasting) pattern could not be replicated by others (e.g., Budd, Barry, Gordon, Rennie, & Michie, 1998). See also Jacobsen and Schröger (2003), who were able to rule out modality-specific refractoriness effects using a stimulus onset asynchrony of 900 msec only.

6. 

Note that tactile N90 amplitudes might have been further modulated by sensory refractoriness effects in the present experiment.

7. 

This is in line with the RT pattern observed in the present study, where discriminating features in Experiment 2 (overall RT of 571 msec) were more time-consuming than discriminating modalities in Experiment 1 (overall RT of 485 msec). See also Müller, Krummenacher, and Heller (2004) and Müller et al. (1995), who showed that gaining explicit knowledge about the target-defining dimension and feature in cross-dimensional search requires extra processing time over and above simple detection.

8. 

We thank an anonymous reviewer for pointing this out to us.

REFERENCES

REFERENCES
Ahissar
,
M.
, &
Hochstein
,
S.
(
2004
).
The reverse hierarchy theory of visual perceptual learning.
Trends in Cognitive Sciences
,
8
,
457
464
.
Budd
,
T. W.
,
Barry
,
R. J.
,
Gordon
,
E.
,
Rennie
,
C.
, &
Michie
,
P. T.
(
1998
).
Decrement of the N1 auditory event-related potential with stimulus repetition: Habituation vs. refractoriness.
International Journal of Psychophysiology
,
31
,
51
68
.
Cohen
,
R.
, &
Rist
,
F.
(
1992
).
The modality shift effect. Further explorations at the crossroads.
Annals of the New York Academy of Sciences
,
658
,
163
181
.
Dyson
,
B. J.
, &
Quinlan
,
P. T.
(
2002
).
Within- and between-dimensional processing in the auditory modality.
Journal of Experimental Psychology: Human Perception and Performance
,
28
,
1483
1498
.
Eimer
,
M.
, &
Van Velzen
,
J.
(
2002
).
Crossmodal links in spatial attention are mediated by supramodal control processes: Evidence from event-related brain potentials.
Psychophysiology
,
39
,
437
449
.
Farah
,
M. J.
,
Wong
,
A. B.
,
Monheit
,
M. A.
, &
Morrow
,
L. A.
(
1989
).
Parietal lobe mechanisms of spatial attention: Modality-specific or supramodal?
Neuropsychologia
,
27
,
461
470
.
Found
,
A.
, &
Müller
,
H. J.
(
1996
).
Searching for unknown feature targets on more than one dimension: Investigating a “dimension-weighting” account.
Perception & Psychophysics
,
58
,
88
101
.
Gehring
,
W. J.
,
Goss
,
B.
,
Coles
,
M. G. H.
,
Meyer
,
D. E.
, &
Donchin
,
E.
(
1993
).
A neural system for error detection and compensation.
Psychological Science
,
4
,
385
390
.
Gondan
,
M.
,
Lange
,
K.
,
Rösler
,
F.
, &
Röder
,
B.
(
2004
).
The redundant target effect is affected by modality switch costs.
Psychonomic Bulletin & Review
,
11
,
307
313
.
Gramann
,
K.
,
Töllner
,
T.
,
Krummenacher
,
J.
,
Müller
,
H. J.
, &
Eimer
,
M.
(
2007
).
Brain electrical correlates of dimensional weighting: An ERP study.
Psychophysiology
,
44
,
277
292
.
Hanewinkel
,
R.
, &
Ferstl
,
R.
(
1996
).
Effects of modality shift and motor response shift on simple reaction time in schizophrenia patients.
Journal of Abnormal Psychology
,
105
,
459
463
.
Hochstein
,
S.
, &
Ahissar
,
M.
(
2002
).
View from the top: Hierarchies and reverse hierarchies in the visual system.
Neuron
,
36
,
791
804
.
Jacobsen
,
T.
, &
Schröger
,
E.
(
2003
).
Measuring duration mismatch negativity.
Clinical Neurophysiology
,
114
,
1133
1143
.
Kleinsorge
,
T.
(
1999
).
Response repetition benefits and cots.
Acta Psychologica
,
103
,
295
310
.
Lamme
,
V. A. F.
, &
Roelfsema
,
P. R.
(
2000
).
The distinct modes of vision offered by feedforward and recurrent processing.
Trends in Neurosciences
,
23, 11
,
571
579
.
Levit
,
R.
,
Sutton
,
S.
, &
Zubin
,
J.
(
1973
).
Evoked potential correlates of information processing in psychiatric patients.
Psychological Medicine
,
3
,
487
494
.
Loveless
,
N. E.
(
1983
).
The orienting response and evoked potentials in man.
In D. Siddle (Ed.),
Orienting and habituation: Perspectives in human research
, (pp.
71
101
).
London
:
John Wiley
.
Maljkovic
,
V.
, &
Nakayama
,
K.
(
1994
).
Priming of pop-out: I. Role of features.
Memory & Cognition
,
22
,
657
672
.
Manuzza
,
S.
(
1980
).
Cross-modal reaction time and schizophrenic attentional deficit.
Schizophrenia Bulletin
,
6
,
654
675
.
Mortier
,
K.
,
Theeuwes
,
J.
, &
Starreveld
,
P. A.
(
2005
).
Response selection modulates visual search within and across dimensions.
Journal of Experimental Psychology: Human Perception and Performance
,
31
,
542
557
.
Müller
,
H. J.
,
Heller
,
D.
, &
Ziegler
,
J.
(
1995
).
Visual search for singleton feature targets within and across feature dimensions.
Perception & Psychophysics
,
57
,
1
17
.
Müller
,
H. J.
, &
Krummenacher
,
J.
(
2006
).
Locus of dimension weighting: Pre-attentive or post-selective?
Visual Cognition
,
14
,
490
513
.
Müller
,
H. J.
,
Krummenacher
,
J.
, &
Heller
,
D.
(
2004
).
Dimension-specific inter-trial facilitation in visual search for pop-out targets: Evidence for a top–down modulable visual short-term memory effect.
Visual Cognition
,
11
,
577
602
.
Müller
,
H. J.
, &
O'Grady
,
R.
(
2000
).
Object-based selection is subject to domain-based attentional limitations.
Journal of Experimental Psychology: Human Perception and Performance
,
26
,
1332
1351
.
Müller
,
H. J.
,
Reimann
,
B.
, &
Krummenacher
,
J.
(
2003
).
Visual search for singleton feature targets across dimensions: Stimulus- and expectancy-driven effects in dimensional weighting.
Journal of Experimental Psychology: Human Perception and Performance
,
29
,
1021
1035
.
Näätänen
,
R.
,
Jacobsen
,
T.
, &
Winkler
,
I.
(
2005
).
Memory-based or afferent processes in mismatch negativity (MMN): A review of the evidence.
Psychophysiology
,
42
,
25
32
.
Nelson
,
D. A.
, &
Lassman
,
F. M.
(
1968
).
Effects of intersignal interval on the human auditory evoked response.
Journal of the Acoustical Society of America
,
44
,
1529
1532
.
Pollmann
,
S.
(
2004
).
Anterior prefrontal cortex contributions to attention control.
Experimental Psychology
,
51
,
270
278
.
Pollmann
,
S.
,
Mahn
,
K.
,
Reimann
,
B.
,
Weidner
,
R.
,
Tittgemeyer
,
M.
,
Preul
,
C.
,
et al
(
2007
).
Selective visual dimension weighting deficit after left lateral frontopolar lesions.
Journal of Cognitive Neuroscience
,
19
,
365
374
.
Pollmann
,
S.
,
Weidner
,
R.
,
Müller
,
H. J.
,
Maertens
,
M.
, &
von Cramon
,
D. Y.
(
2006
).
Selective and interactive neural correlates of visual dimension changes and response changes.
Neuroimage
,
30
,
254
265
.
Pollmann
,
S.
,
Weidner
,
R.
,
Müller
,
H. J.
, &
von Cramon
,
D. Y.
(
2000
).
A fronto-posterior network involved in visual dimension changes.
Journal of Cognitive Neuroscience
,
12
,
480
494
.
Pollmann
,
S.
,
Weidner
,
R.
,
Müller
,
H. J.
, &
von Cramon
,
D. Y.
(
2006
).
Neural correlates of dimension weighting.
Visual Cognition
,
14
,
877
897
.
Rist
,
F.
, &
Cohen
,
R.
(
1988
).
Effects of modality shift on event related potentials and reaction times of chronic schizophrenics.
In R. Johnson, J. Rohrbaugh, & R. Parasuraman (Eds.), Current research in event-related brain potentials.,
Amsterdam
:
Elsevier
.
Rodway
,
P.
(
2005
).
The modality shift effect and the effectiveness of warning signals in different modalities.
Acta Psychologica
,
120
,
199
226
.
Roeber
,
U.
,
Berti
,
S.
,
Widmann
,
A.
, &
Schröger
,
E.
(
2005
).
Response repetition vs. response change modulates behavioral and electrophysiological effects of distraction.
Cognitive Brain Research
,
22
,
451
456
.
Spence
,
C.
,
Nicholls
,
M.
, &
Driver
,
J.
(
2001
).
The cost of expecting events in the wrong sensory modality.
Perception & Psychophysics
,
63
,
330
336
.
Tiitinen
,
H.
,
May
,
P.
,
Reinikainen
,
K.
, &
Näätänen
,
R.
(
1994
).
Attentive novelty detection in humans is governed by pre-attentive sensory memory.
Nature
,
372
,
90
92
.
Töllner
,
T.
,
Gramann
,
K.
,
Müller
,
H. J.
,
Kiss
,
M.
, &
Eimer
,
M.
(
2008
).
Electrophysiological markers of visual dimension changes and response changes.
Journal of Experimental Psychology: Human Perception and Performance
,
34
,
531
542
.
Töllner
,
T.
,
Zehetleitner
,
M.
,
Gramann
,
K.
, &
Müller
,
H. J.
(
submitted
).
Top–down weighting of visual dimensions: Behavioural and electrophysiological evidence.
Verleger
,
R.
, &
Cohen
,
R.
(
1978
).
Effects of certainty, modality shift and guess outcome on evoked potentials and reaction times in chronic schizophrenics.
Psychological Medicine
,
8
,
81
93
.
Vogel
,
E. K.
, &
Luck
,
S. J.
(
2000
).
The visual N1 component as an index of a discrimination process.
Psychophysiology
,
37
,
190
203
.
Wang
,
Y.
,
Cui
,
L.
,
Wang
,
H.
,
Tian
,
S
, &
Zhang
,
X.
(
2004
).
The sequential processing in visual feature conjunction mismatch in the human brain.
Psychophysiology
,
41
,
21
29
.
Weidner
,
R.
,
Pollmann
,
S.
,
Müller
,
H. J.
, &
von Cramon
,
D. Y.
(
2002
).
Top–down controlled visual dimension weighting: An event-related fMRI study.
Cerebral Cortex
,
12
,
318
328
.
Wolfe
,
J. M.
(
1994
).
Guided Search 2.0: A revised model of visual search.
Psychonomic Bulletin & Review
,
1
,
202
238
.