People typically respond faster to a stimulus when it is accompanied by a task-irrelevant accessory stimulus presented in another perceptual modality. However, the mechanisms responsible for this accessory-stimulus effect are still poorly understood. We examined the effects of auditory accessory stimulation on the processing of visual stimuli using scalp electrophysiology (Experiment 1) and a diffusion model analysis (Experiment 2). In accordance with previous studies, lateralized readiness potentials indicated that accessory stimuli do not speed motor execution. Surface Laplacians over the motor cortex, however, revealed a bihemispheric increase in motor activation—an effect predicted by nonspecific arousal models. The diffusion model analysis suggested that accessory stimuli do not affect parameters of the decision process, but expedite only the nondecision component of information processing. Consequently, we conclude that accessory stimuli facilitate stimulus encoding. The visual P1 and N1 amplitudes on accessory-stimulus trials were modulated in a way that is consistent with multisensory energy integration, a possible mechanism for this facilitation.
During most everyday activities, people receive information from multiple sensory modalities. When you ride your bicycle through a city center, for example, you see the road and the traffic around you, hear cars approaching from behind, and feel the pedals and steering wheel of your bicycle. The signals from the different modalities are not processed independently, but are integrated into coherent representational states. Cognitive psychologists have demonstrated multisensory integration in several psychological phenomena. In ventriloquism, for example, the source of an auditory signal is wrongfully perceived at the location of a visual cue (Howard & Templeton, 1966). Multisensory processing can also lead to a change in the perceived signal itself. This was illustrated in a classic experiment in which a face articulating “gaga” was presented visually, while “baba” was presented aurally. It was found that people usually combined the signals from the two sources and perceived “dada” (McGurk & MacDonald, 1976). The present article addresses another striking instance of cross-modal interaction: the phenomenon that task-irrelevant stimulation (i.e., noise) in one perceptual modality can speed up responses to stimuli concurrently presented in another perceptual modality.
It has repeatedly been found that responses in reaction time (RT) tasks are shorter when a salient but task-irrelevant accessory stimulus presented in another perceptual modality accompanies the imperative stimulus, compared to when the imperative stimulus is presented alone (e.g., Bernstein, Clark, & Edelstein, 1969a, 1969b). This speed-up of RTs—often without a concomitant increase in errors—has been referred to as the accessory stimulus (AS) effect. The AS effect has been found in both simple and choice RT tasks (e.g., Bernstein et al., 1969a, 1969b; Morrell, 1968), is largest for auditory stimuli accompanying visual imperative stimuli (Bernstein, 1970; Davis & Green, 1969), and increases in size with the intensity of the AS (Stahl & Rammsayer, 2005). Because the AS is typically presented simultaneously with, or in close temporal proximity to, the imperative stimulus, it has no value for the participant as a cue to start voluntary preparation. Indeed, AS effects have been found even when the AS lags the imperative stimulus (e.g., Stahl & Rammsayer, 2005; Bernstein et al., 1969a, 1969b). In addition, in most experiments, the predictive value of the AS is limited by the inclusion of trials on which no AS is presented (no-AS trials), as well as trials on which the AS is not followed by an imperative stimulus (catch trials).
The various explanations of the AS effect that have been proposed so far can be divided into four types of accounts, depending on the components of information processing that are assumed to be affected. One account of the AS effect is that accessory stimuli facilitate stimulus encoding. In particular, it has been proposed that stimulus energy is combined across different modalities in such a way that adding an auditory AS is comparable to increasing the intensity of the visual imperative stimulus (Bernstein, Rose, & Ashe, 1970). According to the energy integration hypothesis, the increased strength of the joint event speeds up the stimulus encoding process, resulting in shorter RTs. The critical assumption of the energy integration hypothesis is supported by the finding that auditory stimuli can increase the perceived intensity of simultaneously presented visual stimuli (Stein, London, Wilkinson, & Price, 1996). At the neural level, an AS effect on stimulus encoding might be explained in terms of the effects of multisensory neurons—neurons that respond to stimuli from more than one modality. Such neurons exist not only in higher-order association areas but also in low-level, modality-specific sensory areas (Ghazanfar & Schroeder, 2006), supporting the notion that multisensory interactions can influence early sensory processing.
According to the second and third accounts, accessory stimuli affect a critical parameter of the decision process that is based on the sensory evidence obtained during stimulus encoding. The mechanism underlying two-choice decisions is well described by the accumulation of noisy information from a stimulus over time (Gold & Shadlen, 2007; Smith & Ratcliff, 2004). Information accumulates toward one or the other of two decision thresholds until one of the thresholds is reached; then the response associated with that threshold is initiated. One possibility is that accessory stimuli speed up the rate with which evidence is accumulated in the decision process (Hackley & Valle-Inclán, 1999), for example, through an AS-triggered, rapid and transient increase in attention to the imperative stimulus. Another possibility is that accessory stimuli do not change the rate of information build-up but instead cause a lowering of the decision threshold (Posner, 1978). According to this view, decisions are made on the basis of less evidence, resulting in shorter RTs and, possibly, more errors. Such a speed–accuracy tradeoff has indeed been found in some AS studies (e.g., Posner, 1978).
The fourth account of the AS effect holds that accessory stimuli speed up motor execution processes. Apparent support for a motoric locus of the AS effect has come from studies that have found an increased response force (Stahl & Rammsayer, 2005; Miller, Franz, & Ulrich, 1999) or a speeding of reflexes (Low, Larson, Burke, & Hackley, 1996; Stafford & Jacobs, 1990) to stimuli accompanied by an acoustic AS. Other evidence that has been presented as support for the motor account is the interaction effect on RT of AS presence and some factors known to affect motor processes, such as tonic muscle tension (Schmidt, Gielen, & van den Heuvel, 1984; Sanders, 1980). Sanders (1980, 1983) has argued, using additive factors logic (Sternberg, 1969), that such interactions indicate a motoric locus for the AS-triggered speeding of RTs. A discussion of the problems with this argument will be deferred until the General Discussion.
Despite a substantial empirical database, there is no general agreement among researchers on which of these four accounts explains most of the data. One possible source of confusion in the debate may be that the various accounts are not mutually exclusive, hence, different portions of the database may be explained by different accounts. Another reason for the lack of agreement may be that it is hard to distinguish the various accounts on the basis of behavioral performance measures alone.
Probably the most conclusive evidence to date has been reported by Hackley and Valle-Inclán (1998, 1999). These investigators recorded the electroencephalogram (EEG) from participants performing an AS task and computed the lateralized readiness potential (LRP) to investigate the timing of the AS effect. The LRP is an EEG index of hand-specific response preparation. It is computed as the difference in EEG activity over the motor cortices contralateral and ipsilateral to the responding hand, and averages zero until the accumulated evidence at the level of the motor cortex for one of the response options is outweighing that for the other response option. Thus, the onset of the LRP reflects the point during the decision process during which, on average, stimulus-specific accumulators have gathered evidence favoring one of the two response options, and this evidence has been transmitted, first to brain areas representing the relevant stimulus–response mappings, and then to the motor cortex where it is expressed in asymmetrical activity of the two hemispheres (Gold & Shadlen, 2007; Spencer & Coles, 1999). Hackley and Valle-Inclán found that accessory stimuli shorten the interval between stimulus onset and LRP onset but not the interval between LRP onset and the overt response. This is strong evidence against the notion that accessory stimuli speed up motor execution processes, and in support of the view that the AS effect develops during stimulus encoding and/or an early phase of the decision-making process (i.e., before the motor cortex begins to reveal the outcome of the decision).
Despite the knowledge gained by these LRP studies, several important questions remain unanswered. For example, is it possible to reconcile the conclusion that accessory stimuli do not speed up motor execution processes with findings of an AS effect on voluntary response force and the amplitude of somatic reflexes? Can we distinguish the possibilities, suggested by LRP studies, that the AS effect develops during stimulus encoding or during an early phase of the decision-making process? And can we find indications that the AS effect is a result of energy convergence in low-level sensory brain areas? We addressed these and other questions in the two experiments reported below.
EXPERIMENT 1: ELECTROPHYSIOLOGY
The aims of this experiment were threefold. First, we tried to replicate the finding, reported by Hackley and Valle-Inclán (1998, 1999), that auditory accessory stimuli speed up visual information processing before LRP onset but not after LRP onset. As noted, this type of information provides important clues about the processing components influenced by accessory stimuli.
Second, we wanted to investigate whether accessory stimuli have an effect on central motor processes that is not revealed by the LRP methodology used in previous research. Specifically, the LRP is a relative measure, which shows the difference in activity between the contralateral and ipsilateral motor cortices, but not the respective activities of each individual motor cortex. Therefore, the LRP does not reveal potential AS-induced nonspecific increases in motor activity—increases in activity that are equal for the contralateral and ipsilateral motor cortices, and that are not expressed in a RT benefit. The possibility that accessory stimuli increase bilateral motor activity without speeding the actual response execution is consistent with proposals that energy-related stimulus properties (e.g., the intensity of the AS) have nonspecific arousal effects that are dissociable from the effects of translating the stimulus into the appropriate response (Sanders, 1983). It is possible to assess the activity of each individual motor cortex by estimating the surface Laplacians over the primary motor areas by means of the source-derivation method (Hjorth, 1975). The Laplacian acts as a high-pass spatial filter by reducing the common activities between neighboring electrodes. It removes the blurring effect of current diffusion through the highly resistive skull, and is considered to give a good approximation of the corticogram (Gevins, 1989).
The third aim of Experiment 1 was to evaluate a prediction of the energy integration hypothesis by examining the effect of accessory stimuli on the P1 and N1, two early visual-evoked ERP components recorded over the lateral occipital cortex. Previous research has shown that these components increase in amplitude with increasing stimulus brightness (i.e., energy; Blenner & Yingling, 1993). Therefore, if auditory accessory stimuli increase energy in brain areas specialized in visual processing, this energy increase (i.e., the converged energy from the visual and auditory stimuli) should manifest in increased amplitudes of the P1 and N1 associated with the visual imperative stimuli. A failure to find such amplitude enhancements would provide evidence against the energy integration hypothesis. It is important to note that the observation of such enhancements, although consistent with the energy integration hypothesis, would not present definitive evidence for this hypothesis; although the use of surface Laplacians improves estimates of the orientation and location (i.e., biased towards superficial sources) of intracerebral generators, this method does not solve the inverse problem. That is, it cannot exclude the possibility that the amplitude increases reflect the summation at the scalp of electrical activity from two or more different cell populations, rather than the summed activity from one source in visual areas. Nevertheless, the current results will be valuable as a basis for future studies designed to distinguish these possibilities. We also compared the latencies of the P1 and N1 components on AS trials and no-AS trials to determine to what extent the AS effect was already present at the corresponding stages of information processing.
Thirteen volunteers participated (10 women; 12 right handed; aged 18–30 years; mean age = 21.5 years). All participants reported normal hearing and normal or corrected-to-normal vision. All participants gave informed consent and received either €15 or course credits for participation.
Stimuli and Procedure
The task used was a slightly modified version of the task used by Hackley and Valle-Inclán (1998). On most trials, the single letter “S” or “T” was presented for 250 msec in the center of the screen. The letter subtended either 1.0° or 0.8° in visual angle, on 80% and 20% of the trials, respectively. When a 1.0° letter was presented, participants were to indicate whether it was an S or a T by pressing a left or a right key (go trials). The key assignment was balanced across participants. When a 0.8° letter was presented, the response was to be withheld (no-go trials). On a randomly chosen 50% of the trials, an AS (800 Hz, 80 dB, 150 msec long tone) was presented 30 msec prior to the letter onset. The tones were presented binaurally through Epymotic air-pulse earphones. Intertrial intervals were 2, 3, or 4 sec. Unlike in Hackley and Valle-Inclán's task, we also included trials on which the AS was presented alone (catch trials). These catch trials were included to discourage premature responses to the AS, and to be able to compare ERPs to auditory-only, visual-only, and combined visual-and-auditory stimulation. Key-press responses were made with the left and the right thumb, and participants were instructed to respond as fast as possible. An ERROR message of 1 sec was displayed following incorrect go trial responses and responses on no-go trials.
Participants completed one practice block, followed by 15 experimental blocks. Ten of the experimental blocks contained 40 go trials, 10 no-go trials, and 6 catch trials each. In order to obtain enough catch trials, the remaining five blocks contained 16 go trials, 4 no-go trials, and 28 catch trials each. These blocks were presented as the 3rd, 6th, 9th, 12th, and 15th blocks of the experiment. After each block the mean RT appeared on the screen, and participants could take a short break if needed. A total of 800 trials was presented throughout the experiment, which lasted about one hour.
Instrumentation and Recording
Visual stimuli were presented on a 19-in. computer monitor, located at a distance of about 60 cm from the participant. Presentation of the visual and auditory stimuli was controlled by a personal computer using E-prime 1.1. EEG was recorded from 64 Ag/AgCl scalp electrodes mounted in an elastic cap, and from the left and right mastoids, using a 64-channel Biosemi active electrode recording system (sampling rate 512 Hz). Two additional electrodes (Common Mode Sense [CMS] and Driven Right Leg [DRL]) were used as reference and ground (see www.biosemi.com/faq/cms&drl.htm for details). The signal was referenced off-line to the average mastoid signal. The horizontal and vertical electrooculogram (EOG) were measured using bipolar recordings from electrodes placed approximately 1 cm lateral of the outer canthi of the two eyes and from electrodes placed approximately 1 cm above and below the participant's left eye. EEG and EOG were high-pass filtered at 0.1 Hz and low-pass filtered at 30 Hz. Electromyographic (EMG) activity of the flexor pollicis brevis was recorded with paired electrodes fixed about 2 cm apart on the skin of the thenar eminence of each hand, band-pass filtered (10–256 Hz), and full-wave rectified.
Signal Processing and Data Analyses
Single-trial epochs were extracted off-line for a period from 500 msec before until 800 msec after the critical event. Ocular and eyeblink artifacts were corrected using the method of Gratton, Coles, and Donchin (1983). Epochs with other artifacts (spike artifacts [50 μV/2 msec] and slow drifts [200 μV/200 msec]) were also discarded. Then, for each participant and each condition of interest, the EEG epochs were averaged with respect to letter onset (imaginary letter onset on catch trials) and EMG onset to create stimulus-locked and EMG-locked averages. A baseline, computed as the average signal activity across the 200 msec prior to the AS, was subtracted for each ERP. The EMG traces were visually inspected and the EMG onsets were hand-scored by an experimenter. We used this method because visual inspection is more accurate than automated algorithms (Hodges & Bui, 1996; Van Boxtel, Geraats, Van den Berg-Lenssen, & Brunia, 1993). To prevent subjective influence on the onset scoring, the experimenter who scored the onsets was unaware of the trial types to which the EMG traces corresponded.
Trials were excluded from the data analyses if the RT was shorter than 100 msec or longer than 1000 msec, or when the response was incorrect. This resulted in the exclusion of 1.4% of the trials. The EMG onset was used to divide the total RT in premotor time (interval between stimulus onset and EMG onset) and motor time (interval between EMG onset and overt response). For the LRP analysis, we used the same procedure as Hackley and Valle-Inclán (1998): Stimulus- and EMG-locked LRPs were computed from monopolar recordings over C3 and C4 using the standard double subtraction method. LRP latency was assessed at 30%, 50%, and 70% of the peak amplitude using jackknife tests (Miller, Patterson, & Ulrich, 1998). For the surface Laplacian estimation, we used the spherical spline interpolation algorithm of Perrin, Pernier, Bertrand, and Echallier (1989), as implemented in Brain Vision Analyzer. This method is based on the entire electrode array and consists of two steps: First, the values recorded at each electrode are interpolated, and then the spatial second derivative of this function is computed. We used 4 as the degree of spline and 10° as the maximum Legendre polynomial. The P1 amplitude was defined as the peak amplitude of the average surface Laplacian over electrodes PO7 and PO8 in the 60–140 msec time window. The N1 amplitude was defined as the peak amplitude of the average surface Laplacian over electrodes P7 and P8 in the 100–200 msec time window.
In agreement with the findings of Hackley and Valle-Inclán (1998), RT on go trials was shorter on AS trials (mean = 501 msec, SD = 77 msec) than on no-AS trials [mean = 519 msec, SD = 80 msec; t(12) = 5.0, p < .001]. [We verified that this AS effect was of a similar magnitude in the blocks with a high probability of catch trials (21 msec) and the blocks with a low probability of catch trials (18 msec), F < 1.] Accuracy on go trials did not differ between AS trials and no-AS trials [97.2% vs. 97.5%; t(12) = 0.6, p = .53]. The percentage of no-go errors (false alarms) was higher on AS trials than on no-AS trials [9.6% vs. 6.8%; t(12) = 2.2, p = .047].
Responses on catch trials were very rare: One of the participants responded to a catch trial twice, whereas the other participants never responded to a catch trial. This indicates that accessory stimuli did not induce fast-guess responses.
Motor and Premotor Time
The premotor time was shorter on AS trials than on no-AS trials [364 msec vs. 379 msec; t(12) = 6.9, p < .001]. The motor time did not differ between AS trials and no-AS trials [122 msec vs. 124 msec; t(12) = 1.4, p = .18].
Figure 1 shows the stimulus- and EMG-locked LRPs for the AS trials and no-AS trials. Consistent with Hackley and Valle-Inclán's (1998) results, we found an AS effect on the stimulus-locked but not on the EMG-locked LRP latency. The difference on the stimulus-locked LRP latency was 16 msec for the 30% amplitude point [t(12) = 0.62, p = .27], 23 msec for the 50% amplitude point [t(12) = 1.65, p = .06], and also 23 msec for the 70% amplitude point [t(12) = 2.04, p = .03]. It is interesting that these effect sizes roughly correspond to the AS effect on RT. In contrast, the EMG-locked LRPs for the AS trials and no-AS trials almost overlapped, and no significant AS effect was found for any of the three time points (all ts < 0.2). Taken together, this pattern of results indicates that, like RT, the LRP onset occurred earlier and was somewhat less variable in latency on AS trials than on no-AS trials. Importantly, accessory stimuli did not speed processes that followed LRP onset.
The Laplacian waveforms over the motor cortex contralateral and ipsilateral to the involved hand are shown in Figure 2, separately for the AS trials and no-AS trials. On AS trials, two early peaks were observed that were absent on no-AS trials. These peaks reflect tone-related activation in the Sylvian fissure, volume-conducted to the vertex (e.g., Giard et al., 1994). Preceding EMG onset, a negative wave developed over the contralateral motor cortex and a positive wave over the ipsilateral motor cortex. This pattern has also been reported in previous studies, and is thought to reflect the activation of the involved motor cortex and the suppression of activation in the noninvolved motor cortex (Miller, 2007; Burle, Vidal, Tandonnet, & Hasbroucq, 2004; Tandonnet, Burle, Vidal, & Hasbroucq, 2003; Vidal, Grapperon, Bonnet, & Hasbroucq, 2003). Importantly, both the ipsilateral and the contralateral waves were more negative in amplitude on AS trials, suggesting that accessory stimuli induced a nonspecific (i.e., bilateral) increase in activation of the motor cortex. At the time of EMG onset, the AS effect on the Laplacian amplitude was 4.2 μV/cm2 for the contralateral (involved) motor cortex and 2.6 μV/cm2 for the ipsilateral (noninvolved) motor cortex.1 A repeated-measures ANOVA with laterality (ipsi/contra) and AS presence as within-subject factors yielded main effects of laterality [F(1, 11) = 60.0, p < .001] and AS presence [F(1, 11) = 9.5, p = .01], but no significant interaction [F(1, 11) = 0.6, p = .47]. Follow-up contrasts indicated that the contralateral negativity, reflecting the activation of the involved motor cortex, was larger on AS trials than on no-AS trials [17.5 μV/cm2 vs. 13.3 μV/cm2; t(11) = 2.2, p = .046]. Likewise, the ipsilateral positive wave, reflecting the inhibition of the noninvolved motor cortex, was smaller in amplitude on AS trials than on no-AS trials [14.5 μV/cm2 vs. 17.1 μV/cm2; t(11) = 2.4, p = .03]. These results confirm the notion that accessory stimuli caused a nonspecific increase in motor cortex activation.
To test the prediction suggested by the energy integration hypothesis, we tested whether accessory stimuli increased the amplitudes of early visual ERP components. More specifically, we assessed the AS effect on the stimulus-locked Laplacian components corresponding to the P1 (electrodes PO7/8) and the N1 (P7/8; see Figure 3). Consistent with the energy integration hypothesis, the P1 amplitude was larger on AS trials than on no-AS trials [t(12) = 4.4, p < .001]. The N1 amplitude was also larger on AS trials, but this effect just missed significance [t(12) = 1.6, p = .065]. Interestingly, as illustrated in Figure 3 (top), the P1/N1 amplitude differences between AS trials and no-AS trials were similar to the amplitudes of the P1 and N1 components elicited by the accessory stimuli on catch (i.e., auditory-only) trials. To further illustrate this, Figure 3 (bottom) shows the waveforms on AS trials (combined visual and auditory), as well as the sum waveform created by adding the waveforms associated with catch trials (auditory-only) and no-AS trials (visual-only). Although they do not entirely overlap, the similarity of these waveforms is remarkable, and consistent with the energy integration hypothesis.2
To assess whether the AS latency effect observed for the RTs and LRPs is already present at the time of the P1 and N1 components, we determined the AS effect on the peak latencies of these components. There was no AS effect on the P1 latency [t(12) = 0.1, p = .46]. The N1 peaked 6 msec earlier on AS trials than on no-AS trials, a small but consistent difference [t(12) = 1.9, p = .04].
The principal findings of Experiment 1 may be summarized as follows. In accordance with previous studies (Hackley & Valle-Inclán, 1998, 1999), we found that the AS effect was entirely confined to the time period prior to LRP onset (∼100 msec prior to EMG onset). Consistent with this finding, the AS effect was reflected in premotor times but not in motor times. A small portion (about one third) of the effect was already apparent 160 msec after stimulus onset, at the time of the N1 peak. Accordingly, most of the effect must have developed between the N1, a component associated with stimulus encoding, and LRP onset, the moment at which the motor cortex begins to reveal the outcome of the decision-making process. These findings confirm that accessory stimuli do not expedite response execution; they indicate that the AS effect reflects a speed-up of stimulus encoding or an early phase of the decision-making process (presumably in association cortices; Gold & Shadlen, 2007). Given that auditory signals can modulate cortical visual processing as early as 40 msec following their onset (Giard & Peronnet, 1999), this temporal “locus” of the AS effect seems consistent with the observation, under some circumstances, of a residual AS effect when the auditory AS lags the imperative stimulus (up to 100 msec; e.g., Stahl & Rammsayer, 2005; Bernstein et al., 1969a, 1969b).
Interestingly, AS trials were associated with increased amplitudes of the P1 and N1 components in a way that is consistent with the energy integration hypothesis. Specifically, the P1/N1 amplitudes on AS trials (combined visual and auditory) were of a similar magnitude as the summed amplitudes observed on no-AS trials (visual-only) and catch trials (auditory-only). Thus, it is possible that the speed-up of RTs on AS trials reflects the effects of energy integration in visual processing areas, a possibility that is consistent with anatomical and physiological findings (Ghazanfar & Schroeder, 2006). However, the data do not rule out an alternative interpretation, namely, that the increased P1/N1 amplitudes reflect the summation at the scalp of signals originating from visual and auditory processing areas. Other methods are necessary to distinguish between these possibilities.
Previous work has found that accessory stimuli increase response force and reflex magnitude, and that, in general, these response-amplitude measures correlate poorly with RT (Stahl & Rammsayer, 2005; Miller et al., 1999; Low et al., 1996). These findings have been viewed as support for the proposal by Sanders (1983) that accessory stimuli trigger a phasic burst of arousal that leads to nonspecific priming of low-level motor pathways, and that this effect occurs independently from the stimulus–response translation processes contributing to RT. Sanders' proposal dovetails nicely with another principal result of the current study—the finding that accessory stimuli evoked a nonspecific (i.e., bilateral) increase in motor cortex activity that, as noted above, was not expressed in an RT benefit. This finding seems to provide direct evidence for an AS-induced nonspecific increase in motor activation, and, furthermore, suggests a possible explanation of why this nonspecific effect is expressed in higher response force (as determined in previous studies; a similar explanation may apply to reflex magnitude) but not in shorter RTs. According to this explanation, response force is determined by the activation of the relevant (i.e., contralateral) motor cortex, which is higher on AS trials. This assumption is consistent with neuroimaging studies and neurophysiological recordings (Maier, Bennett, Hepp-Reymond, & Lemon, 1993; Cramer et al., 1992). In contrast, choice RT is dependent on (or at least scales with) the difference between the activity in the relevant and irrelevant motor cortex, which is not affected, due to the nonspecificity of the AS effect. This assumption is consistent with previous results indicating that the LRP amplitude at the time of EMG onset is constant across spontaneous variations in RT (Gratton, Coles, Sirevaag, Eriksen, & Donchin, 1988; Mordkoff & Grosjean, 2001), and with the present finding that the EMG onsets on AS trials and no-AS trials were associated with the same LRP amplitude. In any case, the assumption is in accordance with an influential class of decision-making models (e.g., Laming, 1968), which assumes that a response is initiated when the difference between the evidence for each of the two possible responses reaches a certain criterion value. One of these models is the diffusion model (Ratcliff, 1978), which will be used in the next study.
EXPERIMENT 2: DIFFUSION MODEL ANALYSIS
In this experiment, we aimed to further clarify which components of information processing are affected by accessory stimuli on the basis of a diffusion model analysis of AS effects on RT and accuracy. The diffusion model is a model of two-choice decision making that defines the decision process as the continuous accumulation of noisy stimulus information over time, from a starting point toward one of two decision criteria or thresholds (Ratcliff, 1978; see Figure 4). When one of the two thresholds is reached, the corresponding response is initiated. There are several reasons to assume that the diffusion model gives an accurate reflection of how the decision process is implemented in the brain. First, the diffusion process is the optimal decision process: It provides the fastest responses for a fixed level of accuracy, or the highest accuracy for a fixed response time (Wald, 1947). Second, the diffusion model explains the dynamics of neuronal activity during decision-making behavior (Gold & Shadlen, 2007; Smith & Ratcliff, 2004). Third, the diffusion model successfully accounts for RT distributions and error rates in a variety of two-alternative forced-choice tasks (e.g., Ratcliff, 2002; Ratcliff, Van Zandt, & McKoon, 1999).
The diffusion model can be helpful in evaluating the various accounts of the AS effect because some of the main model parameters correspond closely to the different processing components emphasized by these accounts. The three most important parameters of the model in this respect are the drift rate, the boundary separation, and the nondecision component. The drift rate (v) is the mean rate of evidence accumulation in the decision process, which depends on the quality of the stimulus and the perceptual system. The higher the absolute value of the drift rate, the faster a decision threshold is reached. If accessory stimuli increase the drift rate of the diffusion model, this would support the idea that accessory stimuli induce a faster build-up of information. The boundary separation (a) is the distance between the two decision criteria. This parameter determines on how much evidence a decision is based, and can be controlled strategically by the decision-maker. If accessory stimuli lower the boundary separation, this would provide support for the notion that the AS effect reflects a lowering of the decision threshold (Posner, 1978). As noted above, a speed–accuracy tradeoff in the empirical data also provides an important diagnostic criterion for a change in decision threshold. Besides the decision process, there are other components of processing involved in a two-choice RT task, namely, stimulus encoding and response execution which, respectively, precede and follow the decision process. In the diffusion model, these nondecision processes are combined into one nondecision component, Ter. A shortening of the nondecision component by accessory stimuli would indicate that stimulus encoding and/or motor execution are speeded.
We applied the diffusion model to data from a standard lexical decision task, in which participants were asked to classify letter strings as a word or a nonword, with task instructions emphasizing reaction speed in half of the blocks and response accuracy in the other half of the blocks. The diffusion model has been shown to provide a good fit of lexical decision data, accounting for the effects of the experimental variables on RTs for correct and error responses, shapes of the RT distributions, and accuracy values (Wagenmakers, Ratcliff, Gomez, & McKoon, 2008; Ratcliff, Gomez, & McKoon, 2004). Importantly, on half of the trials, the letter string was preceded by an auditory AS, and our major aim was to examine which model parameter(s) could best account for the corresponding differences in task performance. In particular, this approach allowed us to test between the two possible interpretations of the AS effect suggested by Experiment 1: speeding of stimulus encoding or speeding of evidence accumulation.
Twenty-one students participated (18 women; 19 right-handed; aged 18–31 years; mean age = 22 years; all native Dutch speakers). All participants reported normal hearing and normal or corrected-to-normal vision. Each participant completed two sessions of approximately 75 min each, on separate days. Participants received either €15 or course credits for participation.
The stimuli were 800 Dutch words and 800 nonwords. Both the words and the nonwords consisted of four, five, or six letters (195 four-letter, 251 five-letter, and 354 six-letter words as well as nonwords). The frequency of the words ranged from 0.07 to 5.48 per million (mean = 3.47, SD = 1.28; Baayen, Piepenbrock, & Gulikers, 1995). The nonwords were generated by replacing one letter of an existing word; vowels were replaced by vowels and consonants by consonants. The words that were used to generate the nonwords were not used as word stimuli.
A 200-msec long, 80-dB, 1000-Hz sine-wave tone was used as the AS. The tones were presented binaurally through headphones.
Participants were tested individually in a dimly lit room. Stimuli were presented on a personal computer screen, with responses collected from the keyboard. On-screen instructions were provided. On most trials a letter string was presented (Courier New font; visual angle = 2.7° for four-letter words and 4.0° for six-letter words), and participants were instructed to decide whether or not each letter string was a Dutch word by pressing the “z” or the “/” key. The key assignment was balanced across participants. The letter string remained on the screen until a response was made, and was followed by an intertrial interval of 2, 3, or 4 sec. On a randomly chosen 50% of the trials, the AS was presented 100 msec prior to the onset of the letter string. Participants were informed that the tones were irrelevant to the task and could be ignored. On 11% of the trials, the AS was presented alone (catch trials) to discourage premature responses to the AS.
In each of the two sessions, participants completed two practice blocks of 27 trials, followed by 20 experimental blocks of 45 trials. Each experimental block consisted of 20 trials on which a letter string was presented alone, 20 trials on which a letter string was presented together with the AS, and 5 catch trials.
Speed–accuracy instructions alternated across blocks. In speed blocks, participants were instructed to respond as quickly as possible, but without making a lot of errors, and responses slower than 750 msec were followed by a message TOO SLOW of 1 sec. When a response was faster than 250 msec, the message TOO FAST was displayed for 1 sec. No accuracy feedback was given in these blocks. In accuracy blocks, participants were instructed to respond as accurately as possible, but without taking more time to respond than necessary, and incorrect responses were followed by a message ERROR of 1 sec. No speed feedback was given in these blocks. Each block started with an on-screen announcement of the upcoming speed–accuracy instruction, which was displayed for 2 sec. At the end of each block, the mean RT and the proportion of correct responses appeared on the screen, and participants could take a short break before initiating the next block.
Figure 5 shows the mean correct RT and mean proportions correct as a function of word type, instruction, and AS presence. RTs smaller than 300 msec or larger than 2500 msec were excluded from analysis, which resulted in the exclusion of 0.5% of the trials. In accordance with previous studies, RTs were shorter on AS trials than on no-AS trials [636 msec vs. 660 msec; F(1, 20) = 75.7, p < .001], yielding a reliable AS effect. Furthermore, RTs were shorter following speed instructions than following accuracy instructions [599 msec vs. 697 msec; F(1, 20) = 42.6, p < .001], and shorter for words than for nonwords [627 msec vs. 669 msec; F(1, 20) = 84.0, p < .001]. AS presence did not interact with instruction (p = .37) or word type (p = .83). However, the latter two variables showed a significant interaction, indicating that the RT difference between the speed and accuracy instructions was larger for nonwords than for words [F(1, 20) = 6.9, p = .016].
Proportion correct showed no reliable difference between AS trials and no-AS trials [both 0.88; F(1, 20) = 1.4, p = .26]. As expected, proportion correct was higher when the instruction emphasized accuracy than when it emphasized speed [0.92 vs. 0.85; F(1, 20) = 44.5, p < .001]. In addition, proportion correct was higher for nonwords than for words [0.90 vs. 0.86; F(1, 20) = 9.8, p = .005]. None of the interactions between the three variables were significant (all ps > .09).
Finally, responses on catch trials were practically absent: One of the participants responded to a catch trial once, whereas the other participants never responded to a catch trial.
Diffusion Model Analysis
For fitting the diffusion model to the data, we used the Diffusion Model Analysis Toolbox (Vandekerckhove & Tuerlinckx, 2007, 2008). The Diffusion Model Analysis Toolbox estimates parameters by maximizing a multinomial likelihood function. The data that are used to fit the diffusion model are the RT distributions for correct and incorrect responses, and the proportion correct responses. To assess the processing components that are affected by accessory stimuli, four different models were fitted to the data. The four models differed with regard to the parameters that were free to vary as a function of AS presence. In one model (the all free model), Ter, a, and v were all left free to vary. In addition, there were three models in which either Ter, a, or v could vary, whereas the other parameters were held constant (the Ter model, a model, and v model, respectively).
The following parameter settings were the same for all models: (1) The intertrial variability in nondecision time (st) was held constant across all conditions; (2) The starting point of the diffusion process (z) was set at a fixed proportion of the boundary separation, such that the bias in starting point was constant across conditions; (3) Boundary separation (a) and the intertrial variability in starting point (sz) were free to vary between the speed and accuracy conditions (Ratcliff & Rouder, 1998, Experiment 1; Ratcliff, Thapar, & McKoon, 2001, Experiment 2); (4) Mean drift rate (v) and intertrial variability in drift rate (η) were free to vary between the word and nonword trials (Ratcliff, Thapar, Gomez, & McKoon, 2004).
The models were fitted to the data in two ways. First, the models were fitted to each participant's data individually. When a participant made 10 or fewer errors in a condition, the participant's error data for this condition were not included in the fitting procedure. Second, the models were fitted to the averaged data. The averaged data were obtained by calculating the accuracy and the RTs for correct and error trials associated with the 0.1, 0.3, 0.5, 0.7, and 0.9 quantiles for each individual participant, and then averaging these values across participants. (Note that the quantile RTs are not the mean RTs within bins [Ratcliff, 1979], but the boundary RTs of each quantile.) The codes that were used to fit the models can be found at users.fmg.uva.nl/ewagenmakers/papers.html.
AS Effects on the Diffusion Model Parameters
To assess which parameters were affected by AS presence, we analyzed the AS effect on the estimates of the Ter, a, and v parameters in the all-free model. Table 1 shows both the average parameter estimates across participants and the parameter estimates resulting from fits of the models to the averaged data. The parameter estimates obtained by the two fitting methods were very similar, which replicates findings from previous studies (e.g., Ratcliff, Gomez, et al., 2004; Ratcliff, Thapar, et al., 2004; Ratcliff et al., 2001). The average parameter estimates across participants and the parameter estimates resulting from fits to the averaged data were within one SD of each other for all parameters. As expected, the boundary separation was smaller when the instruction emphasized speed than when it emphasized accuracy [F(1, 20) = 48.1, p < .001]. In addition, drift rates were higher for words than for nonwords [F(1, 20) = 16.6, p = .001]. Importantly, neither boundary separation nor drift rate was affected by AS presence [both F(1, 20) < 1]. In contrast, the nondecision component, Ter, was significantly smaller on AS trials than on no-AS trials [t(20) = 5.7, p < .001]. These results suggest that accessory stimuli shorten one or more nondecision processes, but do not affect the decision process itself.
|Average Values across Participants|
|Ter||.471 (.027)||.488 (.027)|
|a (speed)||.097 (.018)||.099 (.018)|
|a (accuracy)||.146 (.037)||.148 (.041)|
|v (words)||.404 (.169)||.391 (.128)|
|v (nonwords)||−.331 (.101)||−.313 (.064)|
|Fits to Averaged Data|
|Average Values across Participants|
|Ter||.471 (.027)||.488 (.027)|
|a (speed)||.097 (.018)||.099 (.018)|
|a (accuracy)||.146 (.037)||.148 (.041)|
|v (words)||.404 (.169)||.391 (.128)|
|v (nonwords)||−.331 (.101)||−.313 (.064)|
|Fits to Averaged Data|
To further assess the AS effect on the different model parameters, we tested which model had the best fit to the data. To compare the adequacy of the four models (i.e., the all-free model, Ter model, a model, and v model) in explaining the observed data we used the Bayesian Information Criterion (BIC; Raftery, 1996), a statistical criterion for model selection. The BIC is an increasing function of the residual sum of squares from the estimated model, and an increasing function of the number of free parameters to be estimated. Thus, the best model is the model with the lowest BIC value. In addition, the raw BIC values were transformed to a probability scale, enabling a more intuitive comparison of the probabilities of each model being the best model (Wagenmakers & Farrell, 2004). The transformation of BIC values to probability values consists of three steps. First, for each model i, the difference in BIC with respect to the model with the lowest BIC value is computed [i.e., Δi(BIC)]. Second, the relative likelihood L of each model i is estimated by means of the following transformation: L(MI|data) ∝ exp[−0.5Δi(BIC)], where ∝ stands for “is proportional to.” Last, the model probabilities are computed by normalizing the relative model likelihoods, which is done by dividing each model likelihood by the sum of the likelihoods of all models. Table 2 summarizes the BIC values and probabilities of each of the four models. Again, both the average values across participants and the values resulting from fits of the model to the averaged data are displayed. The Ter model had, by far, the best fit, both for the individually fitted data and for the averaged data. In the individual analyses, the Ter model yielded the best fit for 18 of the 21 participants. For the sake of completeness, we also examined the models in which combinations of two parameters (Ter and a; Ter and v; a and v) were free to vary as a function of AS presence. The BIC values of these three models were all worse than that of the Ter model.
|Average Values across Participants|
|Fits to Averaged Data|
|Average Values across Participants|
|Fits to Averaged Data|
p = BIC model probability.
To examine the RT distributions, the 0.1, 0.3, 0.5, 0.7, and 0.9 quantile RTs of each participant were averaged across participants. Figure 6 shows the mean correct quantile RTs as well as the mean proportions correct in each condition. The predicted quantile RTs and proportions correct from the best fitting model (the Ter model) are indicated as well. Figure 6 shows that all five quantile RTs of the correct responses were shorter on AS trials than on no-AS trials. However, the absolute AS effect was small relative to the differences between the quantile RTs, which makes visual inspection difficult. To examine the AS effect in more detail, we calculated the RT difference between AS trials and no-AS trials (i.e., the AS effect) for each of the five correct RT quantiles. The resulting delta plot provides a way of zooming in on the AS effect at different points of the RT distribution (e.g., Ridderinkhof, 2002; De Jong, Liang, & Lauber, 1994). Figure 7 shows the delta plots for the observed data and for the data produced by the best-fitting Ter, a, and v models. The AS effect is rather constant across the 0.1–0.7 quantiles, as is predicted by the Ter model, but is somewhat increased for the 0.9 quantile. The a and v models both predict that the AS effect gradually increases as RTs become longer. Most of the conditions in the observed data did not show this pattern, which explains why the Ter provided a better account of the data than the a and v models. In addition, an AS effect on a or v would lead to different proportions of correct responses in AS trials and no-AS trials, which was not found in the data.
We applied the diffusion model to the data from a lexical decision experiment in which the visual imperative stimuli (letter strings) were accompanied by an auditory AS or not. The diffusion model analysis of these data provided important evidence regarding the source of the AS effect. The fit of a model in which all critical parameters were left unconstrained showed that the AS effect was largely accounted for by a change in the nondecision component Ter. In contrast, the decision parameters drift rate and boundary separation, although sensitive to other experimental variables, were not affected by AS presence. In the regular behavioral analyses, we also found no indications for an AS effect on boundary separation: There was no speed–accuracy tradeoff between AS trials and no-AS trials, and no interaction between the effects of AS presence and instruction (emphasis on speed or accuracy), a variable which affected boundary separation. A comparison of models in which only one parameter was allowed to vary between AS trials and no-AS trials pointed in the same direction: For almost all of the participants, the Ter model was best able to explain the data. The Ter model was also significantly better than models in which combinations of two parameters or all three parameters were free to vary as a function of AS presence. Finally, the AS effect was relatively constant across the RT distribution. This implies that accessory stimuli did not alter the shape of the RT distribution but shifted the complete distribution to the left, which is consistent with an effect on the nondecision component.
These results strongly suggest that accessory stimuli do not affect the decision process itself, but instead speed up nondecision processes. Based on the diffusion model analysis alone, it cannot be determined whether the shortening of the nondecision component reflects a speeding of stimulus encoding or response execution, or both. However, the electrophysiological results of Experiment 1 and previous work (Hackley & Valle-Inclán, 1998, 1999) rule out a speeding of response execution. Therefore, the combined results from Experiments 1 and 2 suggest that the AS effect reflects speeding of the stimulus encoding process.
We conducted two experiments to assess which components of information processing are affected by accessory stimuli. The combined results of the two experiments have led us to the following three main conclusions. First, accessory stimuli speed up encoding of the imperative stimulus. This is possibly the result of energy integration in visual processing areas. Second, accessory stimuli cause a bilateral (nonspecific) increase in cortical motor activation, which is not expressed in an RT benefit. Third, accessory stimuli have little or no effect on the decision process. Each conclusion will be addressed below.
Accessory Stimuli Speed Up Encoding of the Imperative Stimulus
The EEG results and diffusion model analyses reported here support the stimulus encoding account of the AS effect. The EEG results indicated that some of the effect was already present at the time of the N1 peak, and that most of the effect developed in the interval between the N1 and LRP onset. The diffusion model analyses suggested that the effect occurred before the start of the decision process, which is presumably some tens of milliseconds before LRP onset, which marks the moment when asymmetric evidence accumulation is revealed at the level of the motor cortex. The notion that accessory stimuli speed up stimulus encoding seems consistent with behavioral studies demonstrating that auditory signals, when presented concurrently with the visual imperative stimulus, can facilitate spatial visual search (Van der Burg, Olivers, Bronkhorst, & Theeuwes, 2008) and target detection in rapid serial visual presentation streams (Dalton & Spence, 2007; Vroomen & de Gelder, 2000), and increase the perceived intensity of visual stimuli (Stein et al., 1996). An interesting goal for future research will be to investigate whether these seemingly similar phenomena are indeed caused by a common mechanism.
The energy integration hypothesis has been forwarded as a specific account of how accessory stimuli might speed up stimulus encoding (Bernstein, 1970). According to this hypothesis, stimulus energy is integrated across different modalities in such a way that adding an auditory AS is comparable to increasing the intensity of the visual imperative stimulus. The notion of intermodal energy convergence—even in presumptive unimodal sensory areas—is consistent with the existence of direct connections between the auditory cortex and the primary visual cortex (Rockland & Ojima, 2003; Falchier, Clavagnier, Barone, & Kennedy, 2002), and multisensory neurons in low-level sensory areas, such as auditory-sensitive neurons in the visual cortex (Morrell, 1972). We found that accessory stimuli increased the amplitudes of early ERP components (P1/N1) over visual processing areas in a way that is consistent with the energy integration hypothesis. Previous studies have found that amplitude increases of early visual ERP components are associated with faster target-detection RTs and forward shifts in the perceived onset of visual stimuli (Talsma, Mulckhuyse, Slagter, & Theeuwes, 2007; McDonald, Teder-Sälejärvi, Di Russo, & Hillyard, 2005), suggesting that increased strength of neural activity in the visual cortex speeds up downstream perceptual processing. Thus, accessory stimuli might have led to increased neural activity in the visual cortex (reflected in P1/N1), which in turn might have speeded up subsequent encoding processes. This possibility is consistent with our finding that the first AS-induced increase in ERP amplitude (∼100 msec after stimulus onset, at the time of the P1 peak) preceded the beginning of the latency effect (∼160 msec after stimulus onset, at the time of the N1 peak). However, due to the inherent limitations of EEG methods (i.e., the “inverse problem”), the ERP findings cannot be taken as conclusive evidence for energy integration in visual processing areas. They provide merely a motivation for future research designed to determine the mechanism underlying the AS effect.
The effect of accessory stimuli on stimulus encoding might be related to stochastic resonance in sensory systems. Stochastic resonance is the counterintuitive phenomenon that adding a certain level of noise to a nonlinear system enhances its response to a weak (subthreshold) input signal (Benzi, Sutera, & Vulpiani, 1981). A possible explanation for stochastic resonance in perceptual systems is that the addition of noise pushes subthreshold stimuli across their threshold, resulting in improved detection of the stimuli (Moss, Ward, & Sannita, 2004). Stochastic resonance effects on stimulus detection have also been demonstrated when the signal and the noise were of different modalities (Manjarrez, Mendez, Martinez, Flores, & Mirasso, 2007). Manjarrez et al. (2007) found that continuous auditory noise improved the detection of subthreshold visual stimuli, which was explained by an increased response of multisensory neurons to the converged auditory and visual input. Along similar lines, the joint presentation of imperative and accessory stimuli might cause a faster increase in neural activation in visual processing areas than the imperative stimulus alone, thereby precipitating detection of the imperative stimulus. Whether similar neural mechanisms are indeed involved in the AS effect and stochastic resonance is an interesting question for future research.
The conclusion that accessory stimuli facilitate stimulus encoding may be important for a better understanding of other phenomena reported in the attentional literature. A prominent example is the warning effect, which is also referred to as the temporal preparation effect. In the temporal preparation paradigm, a warning stimulus announces the onset of an imperative stimulus. Unlike in the AS paradigm, the interval (or foreperiod) between warning stimulus and imperative stimulus is long enough to enable deliberate preparation (usually >500 msec). When foreperiods are constant within blocks but vary between blocks, the typical finding is that RT increases with increasing foreperiod length (Niemi & Näätänen, 1981). This is thought to reflect a more difficult estimation of the timing of the imperative stimulus for longer foreperiods (Klemmer, 1956). LRP studies and psychophysical measurements have yielded evidence for a premotoric locus of the effect (Rolke & Hofmann, 2007; Müller-Gethmann, Ulrich, & Rinkenauer, 2003; but see Rudell & Hu, 2001). Furthermore, animal research has indicated that during the foreperiod interval there is a gradual increase in the firing rate of visual neurons (Ghose & Maunsell, 2002), suggesting that the benefit of temporal preparation is at least in part due to perceptual changes. Although the warning effect does not reflect motoric changes, the degree of temporal preparation is known to affect response force (Mattes & Ulrich, 1997) and reflex amplitude (Brunia & van Boxtel, 2000). Thus, in several regards, there is a marked similarity between the effects of temporal preparation and accessory stimulation. Indeed, Bernstein, Chu, Briggs, and Schurman (1973) have suggested that enhanced preparation is one of the mechanisms underlying the AS effect. Although warning stimuli cause a gradual increase in the firing rate of visual neurons, accessory stimuli might cause an immediate increase in firing rate. This would imply that the warning effect and the AS effect correspond to, respectively, endogenous and exogenous instances of the same process (cf. Hackley & Valle-Inclán, 2003).
Accessory Stimuli Cause a Nonspecific Increase in Motor Activation
Besides an effect on stimulus encoding, accessory stimuli induced a bilateral (nonspecific) increase in motor activation, which had no effect on RT. This finding supports the proposal by Sanders (1983) that accessory stimuli trigger a phasic burst of arousal that leads to nonspecific priming of low-level motor pathways, and that this effect occurs independently from the stimulus–response translation processes contributing to RT. It also has important implications for previous findings of AS effects on motor processes. Interactions of AS presence with manipulations that influence motor processes (e.g., instructed tonic muscle tension) have been interpreted, using additive-factors logic, as evidence that accessory stimuli affect the speed of motor processes (Schmidt et al., 1984; Sanders, 1980). One problem with this line of reasoning is that the critical assumptions underlying the additive-factors logic are highly disputed. For example, researchers have challenged the assumption that information processing consists of a sequence of discrete nonoverlapping stages (e.g., Spencer & Coles, 1999). But even setting aside the problems with these assumptions, an interaction between accessory stimulation and motor manipulations only indicates that accessory stimuli influence motor processes; the interaction does not specify the nature of this influence and whether it is associated with a change in the duration of motor processes. An AS-induced nonspecific increase in motor activation, even when having no direct effect on RT, may modulate the effects of other variables on the duration of motor processes (hence, RT), and therefore, could have been responsible for the interactions that were found in studies using additive-factors logic.
As discussed above, the conclusion that accessory stimuli caused a bilateral increase in motor cortex activation also offers an explanation for previous findings that accessory stimuli increase response force, independently from their effects on RT (Stahl & Rammsayer, 2005; Miller et al., 1999). According to this explanation, the AS-induced stronger activation of the relevant (contralateral) motor cortex causes an increase in response force. Conversely, there is no evidence that a bilateral increase in motor activation affects choice RT. Instead, it appears that choice RT is dependent on the difference between the activity in the relevant and irrelevant motor cortex (Gratton et al., 1988; Mordkoff & Grosjean, 2001), which was not substantially affected by accessory stimuli in Experiment 1. It is plausible that the AS-evoked nonspecific arousal effect also increases the excitability of other motor systems. If so, this may explain the finding of an increased photic blink reflex when the reflex-eliciting stimulus was accompanied by an acoustic AS (Low et al., 1996).
Accessory stimuli might activate the motor cortex either directly, via connections between the auditory cortex and the motor cortex (Ermolaeva, Tolchenova, & Brukhanskaya, 1981; Buser & Imbert, 1961), or indirectly. One possible indirect way in which accessory stimuli could activate the motor cortex is via the locus coeruleus, the main noradrenergic nucleus in the brainstem. Locus coeruleus neurons exhibit a rapid increase in activity following motivationally significant or salient stimuli (Aston-Jones, Rajkowski, & Cohen, 2000). This causes the release of norepinephrine in cortical and subcortical projection areas, which increases the responsivity of efferent neurons to their input (Servan-Schreiber, Printz, & Cohen, 1990). It is plausible that the high-intensity auditory accessory stimuli that were used in the current study, by virtue of their salience, caused a phasic locus coeruleus response. The resulting release of norepinephrine may have caused the AS-induced increase in motor activation. In line with this hypothesis, it has been shown that the availability of norepinephrine is critical for an AS-induced increase of the masseteric-reflex amplitude (Stafford & Jacobs, 1990). It remains to be determined whether the noradrenergic system is also involved in AS-induced changes in voluntary motor responses.
Accessory Stimuli Have Little or No Effect on the Decision Process
Our diffusion model analyses suggested that AS presence did not affect the main parameters of the decision process: the rate of evidence accumulation and the decision threshold. In addition, no AS-induced speed–accuracy tradeoff was found in either of the two experiments. These findings suggest that accessory stimuli did not have a substantial effect on the decision process. However, the increased number of no-go errors (i.e., false alarms) suggests that accessory stimuli induced a lowering of the decision threshold for the go/no-go decision (Gomez, Ratcliff, & Perea, 2007). Note that go responses were much more frequent than no-go responses (80% vs. 20%), which probably resulted in a bias toward the go response. In terms of the diffusion model, this means that the starting point for the go–no-go decision was closer to the go threshold than to the no-go threshold. In contrast, the decision on which hand to respond with was unlikely to be biased toward one of the decision thresholds because left and right responses occurred equally often. The effect of a lowering of the decision threshold on the probability that the diffusion process reaches that threshold by mistake is larger as the threshold is closer to the starting point. Therefore, it is possible that accessory stimuli caused a lowering of the decision thresholds that was too small to significantly affect the number of errors in the left–right decision, but large enough to increase the number of incorrect go responses in the go/no-go decision.
This hypothesis predicts that accessory stimuli only induce a speed–accuracy tradeoff in situations in which the decision threshold is close to the starting point of the decision process. Aside from circumstances that induce a strong response bias, this is likely to be the case in easy choice RT tasks. Previously, studies provide strong support for this prediction: Significantly increased error rates on AS trials have generally been found in studies using relatively simple tasks (e.g., requiring a spatially compatible stimulus–response mapping) with very short mean RTs (<350 msec), suggesting that the response threshold was close to the starting point (Low et al., 1996; Schmidt et al., 1984; Posner, Klein, Summers, & Buggie, 1973). In contrast, the absence of a significant AS effect on error rate has been found in more complex tasks that produced intermediate to long mean RTs (>500 msec; e.g., Hackley & Valle-Inclán, 1999; De Jong, 1991, Experiment 1; the present two experiments). To prevent too many errors, the decision thresholds in these more complex tasks were probably at a relatively large distance from the starting point. Thus, previous findings of AS effects on error rates are consistent with the hypothesis that accessory stimuli cause a small lowering of the decision thresholds, which is only expressed in an increased error rate when the threshold is close to the starting point.
Our LRP findings showed that AS presence did not affect the response-locked LRP. According to the continuous flow theory (Eriksen & Schultz, 1979), stimulus evaluation and response activation proceed largely in parallel, and response activation is continuously influenced by the output of the stimulus evaluation process. This suggests that the LRP is an accurate reflection of the accumulated evidence in the decision process, and corresponds to the drift rate in the diffusion model. Although systematic evidence for this view is still missing, important support has been provided by electrophysiological data (Coles, Gratton, & Donchin, 1988; Gratton et al., 1988) and computational considerations (Usher & McClelland, 2001). To the extent that the LRP indexes an evidence accumulation process, the absence of an effect of AS presence on the response-locked LRP suggests that neither the rate of evidence accumulation nor the decision threshold was affected by accessory stimuli. This would be consistent with our diffusion model analyses.
Summary of Conclusions
Our findings suggest that accessory stimuli facilitate encoding of the imperative stimulus. A possible mechanism for this facilitation, consistent with anatomical and physiological findings, is energy integration in visual processing areas. To further investigate this possibility, a closer link with the multisensory integration literature and associated methods is warranted. In addition, we found that accessory stimuli induce a bilateral increase in motor activation that is independent of the RT benefit. This finding provides new and direct support for nonspecific arousal models, and offers an explanation for previously reported AS effects on response-amplitude measures. Finally, we found no evidence that accessory stimuli affect the rate of evidence accumulation in the decision process. An AS-induced lowering of the decision threshold, if present at all, is small, and is translated in increased error rates only for decisions with a starting point that is already close to the decision threshold. We believe that these findings, obtained by a combination of electrophysiology and diffusion model analyses, provide an important contribution to our understanding of the effects of accessory stimuli on information processing. One important aim for future research will be to combine these two methods in a single experiment, such that the various types of results can be more easily integrated.
This research was supported by the Netherlands Organization for Scientific Research. We thank Borís Burle for his advice on the EEG methods, Joachim Vandekerckhove for his advice on the diffusion model analysis, Sander Los for helpful comments on the additive factors logic, and Gilles Dutilh, Sabrina La Fors, and Jonne Oldenburg for their technical assistance.
Reprint requests should be sent to Marieke Jepma, Department of Cognitive Psychology, Leiden University, Wassenaarseweg 52, 2333 AK, Leiden, the Netherlands, or via e-mail: email@example.com.
The analyses reported here controlled for the difference in pre-EMG baseline between AS trials and no-AS trials. This baseline difference reflects the tone-elicited negative component (see Figure 2, top), smeared out in the EMG-locked averages. Thus, we subtracted the baseline, defined as the amplitude of the peak immediately preceding EMG onset, from the Laplacian amplitudes at the time of EMG onset. One participant was excluded from these analyses because he did not show a clear baseline peak.
Most ERP studies on multisensory processing focus on superadditive enhancements (i.e., situations in which the multisensory response exceeds the sum of the unisensory responses) to demonstrate multisensory interactions. Cell recording studies, however, have revealed that superadditivity is merely one facet of multisensory integration, and one that is produced under very specific circumstances, namely, when the unisensory component stimuli are weakly effective. Across the broader range of stimulus intensities, the majority of the multisensory interactions approximate linear summation (i.e., additive enhancements; reviewed in Stanford & Stein, 2007).