Abstract

The need to engage in multiple tasks simultaneously is often encountered in everyday experience, but coordinating between two or more tasks can lead to impaired performance. Typical investigations of multitasking impairments have focused on the performance of two tasks presented in close temporal proximity on discrete trials; however, such paradigms do not match well with the continuous performance situations more typically encountered outside the laboratory. As a result, the stages of information processing that are affected during multisensory continuous dual tasks and how these changes in processing relate to behavior remain unclear. To address these issues, participants were presented simultaneous rapid visual and auditory stimulus sequences under three conditions: attend visual only, attend auditory only, and dual attention (attend both visual and auditory). Performance, measured in terms of response time and perceptual sensitivity (d′), revealed dual-task impairments only in the auditory task. Neural activity, measured by the ERP technique, revealed that both early stage sensory processing and later cognitive processing of the auditory task were affected by dual-task performance, but similar stages of processing of the visual task were not. Critically, individual differences in neural activity at both early and late stages of information processing accurately rank-ordered individuals based on the observed difference in behavioral performance between the single and dual attention conditions. These results reveal relationships between behavioral performance and the neural correlates of both early and late stage information processing that provide key insights into the complex interplay between the brain and behavior when multiple tasks are performed continuously.

INTRODUCTION

Many everyday and job-related skills require continuous performance of multiple tasks that involve multiple sensory modalities. For example, air-traffic controllers must simultaneously and continuously monitor both auditory communications from pilots and visual radar displays. Drivers may choose to hold a cell phone conversation while visually scanning the road for other vehicles. It has been well established that performing multiple tasks simultaneously can result in reduced accuracy and slower RT compared with when those tasks are completed in isolation (for a review, see Pashler, 1994). Dual-task interference has classically been studied using brief punctate trial-based tasks, such as in studies investigating the psychological refractory period (PRP) in which an RT increase is seen when the delay between two discrete tasks is reduced (Telford, 1931). This RT increase is typically interpreted as revealing a central resource limitation or bottleneck at decision or response selection stages of information processing (Zylberberg, Slezak, Roelfsema, Dehaene, & Sigman, 2010; Lien, Ruthruff, & Johnston, 2006; Sigman & Dehaene, 2005; Pashler, 1994; de Jong, 1993; Welford, 1967, but for an alternative account, see Meyer & Kieras, 1997).

Although studies of the PRP phenomenon give insight to the cognitive limitations of performing two successive tasks, the paradigm only imposes dual-task conditions in short, discrete moments in time. Thus, a distinction can be made between discrete dual tasks that are performed in short overlapping or adjacent periods, such as the PRP, and continuous dual tasks in which two tasks are performed simultaneously for extended periods. In contrast to the limitations observed in discrete PRP tasks, studies of continuous dual tasks, such as simultaneously monitoring two visual streams for targets, have reported that RT and accuracy often do not exhibit interference effects relative to single task continuous performance (Allport, Antonis, & Reynolds, 1972). The lack of interference contrasts the effects observed in trial-based paradigms and may be the result of appropriate task scheduling and switching (Pashler, 1994). Nevertheless, there are conditions in which continuous dual tasks can result in clear performance detriments, such as impaired driving while talking on a mobile phone (Just, Keller, & Cynkar, 2008; Strayer, Drews, & Crouch, 2006; Strayer & Johnston, 2001). Auditory tasks, such as word generation or listening to passages that must be committed to memory, can cause performance detriments in simultaneous visual tasks (Gherri & Eimer, 2011; Drews, Pasupathi, & Strayer, 2008) and result in reduced visual processing (Uchiyama et al., 2012). However, dual-task interference can be reduced, or eliminated, if the auditory task is less attentionally demanding (Drews et al., 2008; Kunar, Carter, Cohen, & Horowitz, 2008; Strayer & Johnston, 2001). Altogether, continuous performance of dual tasks does not always result in performance impairment, but shared attentional resources or central processing bottlenecks appear to be responsible for detriments in visual performance when auditory information is simultaneously attended and acted upon.

A complete understanding of the neural mechanisms of continuous dual-task performance requires understanding the stage of information processing that is impaired. To gain insight into this issue, a variety of studies have recorded EEG data during the performance of discrete and continuous dual tasks (Kok, 1997). For example, EEG studies of concurrent visual and auditory stimulation have revealed interference of neural processing in visual tasks (Jacoby, Hall, & Mattingley, 2012; Joassin, Maurage, Bruyer, Crommelinck, & Campanella, 2004; Shams, Kamitani, Thompson, & Shimojo, 2001). ERPs extracted from continuous EEG data have also been used to isolate the stage of neural information processing that is affected by dual-task performance. For instance, studies using discrete tasks such as the attentional blink task, in which a performance impairment occurs when two targets are to be processed within a few hundred milliseconds of each other (Shapiro, Raymond, & Arnell, 1997), and PRP tasks have found that early visual ERP components (e.g., P1, N1) are typically unaffected by dual-task performance (Marti, Sigman, & Dehaene, 2012; Sigman & Dehaene, 2008; Brisson, Robitaille, & Jolicoeur, 2007; Sergent, Baillet, & Dehaene, 2005; Luck, 1998; Vogel, Luck, & Shapiro, 1998). Investigations of visual-evoked ERPs in continuous dual tasks have been less common, although there is evidence that attending to and remembering auditory messages does not affect the visual P1 component evoked during visual search tasks (Gherri & Eimer, 2011).

A number of studies have investigated the modulation of early sensory processing by measuring auditory ERPs in continuous dual-task performance. Of particular relevance is the early auditory N1 component, which is sensitive to selective spatial attention (Woldorff et al., 1993; Hillyard, Hink, Schwent, & Picton, 1973). The amplitude of the auditory N1 is known to scale directly with the amount of priority given to the auditory domain in continuous audiovisual dual tasks at fast presentation rates (Parasuraman, 1985). Additionally, the continuous performance of a primary visual task concurrently with a secondary tone monitoring task results in a reduced N1 amplitude to auditory targets compared with when the auditory monitoring task is performed in isolation, suggesting that the N1 can index the allocation of attention to auditory stimuli under multimodal dual-task conditions (Singhal, Doerfling, & Fowler, 2002).

Decision and response selection stages of information processing during dual-task performance have also been investigated using later ERP components. The most frequently studied component in this regard is the P3, a positive deflection starting about 300-msec poststimulus that increases in strength when task-relevant infrequent auditory or visual stimuli are attended and detected (Duncan-Johnson & Donchin, 1977; Sutton, Braren, Zubin, & John, 1965). Discrete dual-task studies (e.g., using the PRP task) have found delays in the latency of the visual P3, supporting the idea of a central bottleneck or resource limitation at later processing stages (Sigman & Dehaene, 2008; Dell'Acqua, Jolicoeur, Vespignani, & Toffanin, 2005; Arnell, Helion, Hurdelbrink, & Pasieka, 2004; Luck, 1998). The amplitude of the visual P3 is known to be greater when a visual item is attended in isolation compared with when attention is divided (Mangun & Hillyard, 1990), an effect that also occurs when auditory information is attended and processed simultaneously with a visual task (Gherri & Eimer, 2011). Interestingly, the amplitude of the P3 in divided attention tasks is inversely related to the difficulty of the primary task but not the secondary task (Isreal, Chesney, Wickens, & Donchin, 1980), and difficult visual tasks reduce P3 amplitude evoked by auditory tasks (Wester, Bocker, Volkerts, Verster, & Kenemans, 2008). Furthermore, auditory and visual P3 amplitudes track consistently with the graded amount of attention devoted to either simultaneous auditory or visual tasks (de Jong, Toffanin, & Harbers, 2010), which is consistent with a central processing limitation.

When considered together, the extant ERP evidence suggests that dual-task performance typically does not modulate the earliest stages of visual information processing (i.e., <250 msec after stimulus presentation) but may alter early stages of auditory processing. However, later stages of information processing (i.e., >250 msec after stimulus presentation) that are more tightly linked with stimulus categorization and recognition in both modalities can be attenuated and delayed.

This Study

Although the current understanding of the neural mechanisms of dual-task performance has been advanced by ERP studies, there remain open questions about the effects of continuous dual-task performance on both auditory and visual information processing. Two questions are most relevant for this study. First, although a number of studies have investigated the effects of continuous performance dual tasks on neural processing using ERPs, these have tended to focus on later stages of processing in either the visual or auditory domain. Second, although there is some evidence of relationships between neural activity measured with ERPs and behavioral measures in single tasks (e.g., Das, Giesbrecht, & Eckstein, 2010; Gerson, Parra, & Sajda, 2005; Kammer, Lehr, & Kirschfeld, 1999; Mangun & Hillyard, 1990; Kutas, McCarthy, & Donchin, 1977), the effects of continuous dual tasks on behavioral performance and neural activity measured with ERPs have typically been analyzed separately and have not been systematically or quantitatively related. As a result, it remains unclear how modulations of visual and auditory neural processing are interrelated during continuous dual tasks and how these modulations are linked to changes in behavior.

The goal of this study was to address these issues by measuring changes in neural processing between single and dual-task conditions, in both visual and auditory modalities, and relating these directly to the corresponding changes in behavioral task performance. EEG was recorded during an auditory task and a visual task that each required the detection of targets among a continuous stream of distractors. These tasks were performed under three attention conditions: attend visual only, attend auditory only, and attend both visual and auditory (i.e., attend dual). Importantly, the auditory and visual stimulation was identical in each condition, and the only difference between conditions was the attentional instruction. This design allowed us to investigate and compare the neural correlates of continuous auditory and visual dual-task performance on behavior and different stages of information processing indexed with ERPs.

METHODS

Participants

Sixteen healthy participants (10 women) ranging in age from 18 to 44 years (mean = 21.5 years, SD = 5.8 years) were recruited through the University of California, Santa Barbara, on-line recruitment system and received either $20 an hour or course credit for participating. All volunteers had normal or corrected-to-normal vision and provided informed consent before the experiment. The University of California, Santa Barbara, Human Subjects Committee approved all procedures.

Stimuli and Materials

Visual stimuli consisted of 8-bit grayscale images of faces and cars taken from the Max Planck Institute for Biological Cybernetics face database (Troje & Bülthoff, 1996). There were 12 unique images of faces and 12 unique images of cars (half frontal view, half 45° angle), each filtered to match the average power spectrum of all images. Independent white Gaussian noise fields were filtered by this average power spectrum and added to the original 24 face/car images in 10 iterations, resulting in 240 images (Das et al., 2010). Visual stimuli were presented on a 19-in. ViewSonic E90F CRT monitor with a resolution of 1024 × 768 pixels and a refresh rate of 75 Hz. Participants sat 125 cm from the monitor in a darkened radio-frequency shielded chamber. Images subtended a visual angle of approximately 4.57°.

Auditory stimuli consisted of words from the military alphabet recorded using a computerized text-to-speech function. Ten two-syllable auditory stimuli were used (bravo, charlie, delta, echo, kilo, oscar, tango, victor, xray, and yankee), with the average duration of playback equaling 499 msec and a standard deviation of 70 msec. Auditory stimuli were presented through two Dell speakers placed to the left and right of the monitor.

Procedure

A schematic of the stimulus sequence is displayed in Figure 1. Images were presented at fixation at a rate of 2 Hz (duration = 500 msec, ISI = 0 msec). The visual task was designed to be a target search for a rare item, so the probability of a face image was held constant at .10. Auditory stimuli were presented at a slower rate (every six images), and each word occurred with .10 probability. The audiovisual sequence ran uninterrupted for 2-min intervals, each followed by a self-paced break. To avoid eye blink artifacts in the EEG data, a 1-sec blink break was offered every 11 sec during the stimulus stream by presenting a blank screen with the word “Blink” at fixation. Participants were instructed to hold their blinks until this blink break.

Figure 1. 

The stimulus presentation sequence for the visual and auditory tasks. Stimulus presentation was identical across the attend visual, attend auditory, and dual attention conditions. For display purposes, the noise on the images shown in the figure is less than in the actual experiment. For further details on the stimulus presentation, see Methods.

Figure 1. 

The stimulus presentation sequence for the visual and auditory tasks. Stimulus presentation was identical across the attend visual, attend auditory, and dual attention conditions. For display purposes, the noise on the images shown in the figure is less than in the actual experiment. For further details on the stimulus presentation, see Methods.

Each participant performed the task under three attention conditions. In the attend visual condition, participants were instructed to monitor the images for the appearance of a face and to press the spacebar key as soon as a face image was detected. Participants were instructed to ignore the auditory stimuli. In the attend auditory condition, participants were instructed to monitor the auditory stream for a target word and press the return key when the target was recognized. The target word was randomly selected at the outset of the auditory-only task and was held constant throughout the task condition. During the attend auditory condition, participants were additionally instructed to fixate the visual sequence but not to respond to any of the images. Finally, the third condition was the attend dual condition in which participants were instructed to simultaneously monitor the visual and auditory streams for targets. The target word in the dual task was selected randomly and separately from the target selection in the auditory-only task. Participants were instructed to give the auditory and visual tasks equal priority in the dual attention condition. The order of the conditions was randomized and counterbalanced across participants.

It is important to note that the visual and auditory stimuli were identical in all conditions of the task, with the only difference being instructions to focus on one modality over another. A total of 4,800 images (4,320 cars, 480 faces) and 800 auditory words (720 nontarget words, 80 target words) were presented in each condition.

Behavioral Data Analysis

Behavioral performance was measured using RT and sensitivity (d′). RTs were computed as the time between the previous stimulus presentation and the button press. These values were averaged across trials for each participant and condition. To index the level of stimulus decision accuracy in each task, the signal detection theory sensitivity measure of d′ was used (Macmillan & Creelman, 1991; Green & Swets, 1966). First, the hit rate (HR) was calculated by the number of correct target-present responses divided by the total number of targets. Second, the false alarm rate (FA) was calculated as the number of target-present responses given to nontargets relative to the total number of nontargets. The d′ was then calculated by subtracting the z transformations: d′ = z(HR) − z(FA). The resulting d′ measure represents the individual participant's sensitivity in discriminating between signal (i.e., target) and noise (i.e., nontarget) stimuli. To test for differences between conditions for behavioral measures, a 2 (Task modality) × 2 (Attention condition − single and dual conditions) repeated-measures ANOVA was used for both RT and d′. We also report the mean (M), SEM, repeated-measures t statistic (t), probability (p), and effect size (η2) for any direct comparisons (Figure 2).

Figure 2. 

The behavioral results are displayed. (A) Sensitivity (d′) of the target detection responses for the visual and auditory tasks. (B) RT of the target detection response for the visual and auditory tasks. The blue lines represent the behavioral performance under single-task attention conditions (i.e., either attend visual for the visual task or attend auditory for the auditory task), and the red lines represent behavioral performance under the dual attention condition. Error bars represent SEM.

Figure 2. 

The behavioral results are displayed. (A) Sensitivity (d′) of the target detection responses for the visual and auditory tasks. (B) RT of the target detection response for the visual and auditory tasks. The blue lines represent the behavioral performance under single-task attention conditions (i.e., either attend visual for the visual task or attend auditory for the auditory task), and the red lines represent behavioral performance under the dual attention condition. Error bars represent SEM.

EEG Data Acquisition

The EEG was measured for each participant using a BioSemi ActiveTwo system consisting of 32 Ag–AgCl sintered electrodes mounted in an elastic cap and placed according to the International 10–20 system. Additional electrodes were placed at the right and left mastoids, as well as 1 cm lateral to the left and right external canthi (horizontal), and above and below each eye (vertical) for the EOG. Data were sampled at 512 Hz and referenced offline to the average mastoid signal. Trials in which EOG channels showed a change in amplitude of ±75 microvolts or greater were classified as eye blink/movements and were excluded from further analysis.

ERP Data Analysis

To test whether dual attention conditions affected specific ERP components, mean amplitude and peak latency analyses were performed. We chose to focus specifically on the early visual P1 and auditory N1 components to index changes in early perceptual processing. Although the short stimulus presentation (500 msec) in this study makes it difficult to completely capture the full P3 component, we do report additional analyses of amplitude and latency for visual and auditory P3 to index later processing changes. Components were identified by first finding the peak across participants and conditions within classically defined time windows and electrodes (e.g., 100–150 msec at occipital sites for the P1) and centering a 40-msec time window on that peak time point. The resulting windows that were used were 91–131 msec for the P1 and 126–166 msec for the N1. A larger window of 400–600 msec was used for the P3 component because of its more extended and coarse waveform (Mangun & Hillyard, 1990) and was cut off at 600 msec to avoid confounds from the stimulus response of the next trial. Plots of the amplitude topography were then produced for each time window and condition (see Figure 3), and the subset of contiguous electrodes showing the strongest amplitude for that component, independent of stimulus condition, was selected by visual inspection and used to create the averaged ERP waveform. This selection process resulted in the following sets of electrodes: O1, O2, and Oz for the visual P1; Cz, CP1, CP2, and Pz for the visual P3; FC1, FC2, Cz, CP1, and CP2 for the auditory N1; and Cz, CP1, CP2, Pz, P3, and P4 for the auditory P3.

Figure 3. 

Topographic plots of ERP amplitude. (A) The mean amplitudes during the time windows for the visual P1 (91–131 msec) and auditory N1 (126–166 msec) are displayed separately for target and nontarget stimuli within each attention condition. (B) The mean amplitudes during the time windows for the visual and auditory P3 (400–500 msec) are displayed separately for target and nontarget stimuli within each attention condition.

Figure 3. 

Topographic plots of ERP amplitude. (A) The mean amplitudes during the time windows for the visual P1 (91–131 msec) and auditory N1 (126–166 msec) are displayed separately for target and nontarget stimuli within each attention condition. (B) The mean amplitudes during the time windows for the visual and auditory P3 (400–500 msec) are displayed separately for target and nontarget stimuli within each attention condition.

The ERP mean amplitudes were calculated by averaging the amplitude across the chosen time window for the electrode set for each component. Latencies for the ERP components were calculated by measuring the time in milliseconds from stimulus onset to the peak amplitude within the component time window. To test the effect of dual-task attention on ERP component amplitudes and latencies, an ANOVA with three factors for attention condition (attend visual, attend auditory, dual attention) was performed for each component separately for target and nontarget stimuli.

Relating ERPs to Behavioral Performance

To determine if the dual attention condition caused differences in the ERP components that were related to corresponding dual attention differences in behavioral performance, a rank order analysis (Kasper, Elliott, & Giesbrecht, 2012; Das et al., 2010) was performed that is more robust to deviations from linear relationships than the Pearson correlation coefficient. This rank order analysis considered the differences in ERP and behavioral measures between single attention (i.e., attend visual or attend auditory) and dual attention conditions. Rank order analyses were performed by comparing pairs of participants to determine whether the difference in amplitude or latency of an ERP component elicited by targets predicted the difference in behavior for each pair. Thus, rank order accuracy was assigned a value of 1 if an attenuation in ERP amplitude or delay in peak latency from single to dual attention conditions predicted a reduction in d′ or slowing of RT. For example, if participant A had a larger P1 amplitude reduction (single attention − dual attention) than participant B, and participant A also had a larger d′ reduction (single attention − dual attention), that rank ordering was considered accurate. As a result, rank order accuracies above chance (0.50) indicate a positive relationship between behavior and ERP changes, whereas accuracies below chance indicate a negative relationship. The rank order process was done for all possible pairs of participants to create an average accuracy. A jackknife method that left out one participant for each full cycle of comparisons was used to compute standard error (Duda, Hart, & Stork, 2001). Tests of significance were done using an independent measures t test against a random permutation rank accuracy (M = 0.50, SEM = 0.03). Because of the possibility of an inflated type I error rate resulting from multiple comparisons, the false discovery rate (FDR) correction q values are reported along with typical p values (Benjamini & Hochberg, 1995).

RESULTS

The results are presented in four separate sections: The first section contains the results of the analysis of behavioral performance, the second section contains the analysis of early stages of information processing (i.e., <250 msec after stimulus presentation), the third section contains the analysis of later stages of information processing (i.e., >250 msec after stimulus presentation), and the fourth section contains the analysis that relates the dual-task modulations in neural activity measured with ERPs and changes in behavioral performance.

Behavioral Performance

Mean d′ and RT are shown as a function of task and attention condition in Figure 2. Mean d′ was larger in the auditory task when detecting words (M = 3.23, SEM = 0.10) compared with visual task when detecting faces (M = 1.96, SEM = 0.11; F(1, 15) = 87.65, p < .01, η2 = 0.85). No other main effects or interactions for d′ were significant (both Fs(1, 15) < 2.36, p > .14, η2 < 0.14). RTs were faster in the visual task (M = 309.96, SEM = 7.32) compared with the auditory task (M = 379.18, SEM = 11.68; F(1, 15) = 32.12, p < .01, η2 = 0.68). Additionally, RTs were faster under the single attention condition (M = 332.44, SEM = 7.50) compared with the dual attention condition (M = 356.71, SEM = 8.42; F(1, 15) = 23.93, p < .01, η2 = 0.62). These effects were qualified by an interaction between task modality and attention conditions (F(1, 15) = 16.91, p < .01, η2 = 0.53). Post hoc t tests revealed that this interaction was driven by the fact that, on the one hand, the auditory task RTs were faster in the attend auditory condition (M = 357.66, SEM = 11.63) compared with the dual attention condition (M = 400.71, SEM = 13.23; t(15) = 4.88, p < .01), whereas on the other hand, there was no difference between attend visual (M = 307.22, SEM = 7.44) and dual attention (M = 312.71, SEM = 7.64) conditions in the visual task (t(15) = 1.52, p > .15).

Measuring Early Stages of Information Processing

To understand the effects of continuous dual-task performance on early stages of information processing, we isolated the visual P1 and auditory N1 ERP components. Comparisons were made for component amplitude and latency between the three different attention conditions (attend visual, attend auditory, dual attention) separately for the target and nontarget stimuli. These comparisons allow us to address the effects of attention on the neural processing of visual and auditory information. Table 1 shows a summary of the important findings for each of the comparisons made.

Table 1. 

ERP Effects in the Continuous Dual Task

Temporal Processing Stage
ERP Component
Stimulus Type
ERP Measure
Key Result
Early Visual P1 Target Amplitude Reduced in attend auditory vs. dual attention condition 
Latency Delayed in attend auditory condition 
Nontarget Amplitude No differences 
Latency Delayed in attend auditory condition 
Auditory N1 Target Amplitude Enhanced in attend auditory conditiona 
Latency No differences 
Nontarget Amplitude Enhanced in attend auditory vs. attend visual conditiona 
Latency Delayed in attend visual vs. attend auditory condition 
Late Visual P3 Target Amplitude Reduced in attend auditory condition 
Latency No differences 
Nontarget Amplitude No differences 
Latency No differences 
Auditory P3 Target Amplitude Reduced in attend visual condition 
Latency Delayed in dual vs. auditory, dual vs. visual, and auditory vs. visual attention conditionsa 
Nontarget Amplitude No differences 
Latency No differences 
Temporal Processing Stage
ERP Component
Stimulus Type
ERP Measure
Key Result
Early Visual P1 Target Amplitude Reduced in attend auditory vs. dual attention condition 
Latency Delayed in attend auditory condition 
Nontarget Amplitude No differences 
Latency Delayed in attend auditory condition 
Auditory N1 Target Amplitude Enhanced in attend auditory conditiona 
Latency No differences 
Nontarget Amplitude Enhanced in attend auditory vs. attend visual conditiona 
Latency Delayed in attend visual vs. attend auditory condition 
Late Visual P3 Target Amplitude Reduced in attend auditory condition 
Latency No differences 
Nontarget Amplitude No differences 
Latency No differences 
Auditory P3 Target Amplitude Reduced in attend visual condition 
Latency Delayed in dual vs. auditory, dual vs. visual, and auditory vs. visual attention conditionsa 
Nontarget Amplitude No differences 
Latency No differences 

Table 1 shows a summary of the relevant results from the ERP component analyses.

aThose comparisons in which there was an effect of dual-task performance.

Visual Target Evoked Responses

The mean ERP waveforms evoked by the visual stimuli are shown in Figure 4A. The mean amplitude for targets differed significantly as a function of attention condition (F(2, 30) = 3.79, p < .04, η2 = 0.20). Post hoc comparisons revealed that this interaction was driven by smaller overall P1 amplitude in the attend auditory condition (M = 3.29, SEM = 0.49) compared with the dual attention condition (M = 3.98, SEM = 0.58; t(15) = 2.70, p < .02). Direct comparison between the attend visual (M = 3.74, SEM = 0.59) and dual-task conditions indicated that the visual P1 did not differ significantly (t(15) = 1.63, p > .10). Similar to the mean amplitude analyses, the analysis of the P1 peak latencies to targets also revealed an effect of attention (F(2, 30) = 7.24, p < .01, η2 = 0.33), which resulted from a later P1 peak in the attend auditory (M = 118.26, SEM = 1.60) compared with both the attend visual (M = 115.94, SEM = 1.30) and dual attention (M = 115.58, SEM = 1.60) conditions (both ts(15) > 2.69, ps < .02).

Figure 4. 

Average ERP waveforms for visual and auditory stimuli. (A) The visual P1 waveforms are displayed separately for target stimuli on the left and nontarget stimuli on the right. (B) The auditory N1 waveforms are displayed separately for target stimuli on the left and nontarget stimuli on the right. (C) The visual P3 waveforms are displayed separately for target stimuli on the left and nontarget stimuli on the right. (D) The auditory P3 waveforms are displayed separately for target stimuli on the left and nontarget stimuli on the right. Blue lines represent the attend visual condition, green lines represent the attend auditory condition, and red lines represent the dual attention condition. The gray bar represents the time window used for mean amplitude and peak latency analyses for the visual P1 (91–131 msec), auditory N1 (126–166 msec), and the visual and auditory P3 components (400–600 msec).

Figure 4. 

Average ERP waveforms for visual and auditory stimuli. (A) The visual P1 waveforms are displayed separately for target stimuli on the left and nontarget stimuli on the right. (B) The auditory N1 waveforms are displayed separately for target stimuli on the left and nontarget stimuli on the right. (C) The visual P3 waveforms are displayed separately for target stimuli on the left and nontarget stimuli on the right. (D) The auditory P3 waveforms are displayed separately for target stimuli on the left and nontarget stimuli on the right. Blue lines represent the attend visual condition, green lines represent the attend auditory condition, and red lines represent the dual attention condition. The gray bar represents the time window used for mean amplitude and peak latency analyses for the visual P1 (91–131 msec), auditory N1 (126–166 msec), and the visual and auditory P3 components (400–600 msec).

Visual Nontarget Evoked Responses

The amplitude analysis of the P1 evoked by the nontargets revealed no effect of attention (F(2, 30) = 3.29, p > .05, η2 = 0.18). Similar to the analysis of the latencies of the P1 evoked by visual targets, there was a significant effect of attention on the latency of the P1 evoked by nontargets (F(2, 30) = 8.59, p < .01, η2 = 0.36). Direct comparisons revealed that the P1 peaked later in the attend auditory (M = 118.51, SEM = 1.52) than both the attend visual (M = 115.70, SEM = 1.29) and dual (M = 115.82, SEM = 1.33) conditions (both ts(15) > 2.97, ps < .01).

Auditory Target Evoked Responses

Figure 4B shows the average ERPs evoked by the auditory stimuli. Mean amplitude analyses of the auditory N1 to target stimuli revealed a significant effect of attention (F(2, 30) = 9.73, p < .01, η2 = 0.39). Direct comparisons revealed an enhanced (i.e., more negative) N1 in the attend auditory condition (M = −4.89, SEM = 0.65) compared with both the dual attention (M = −2.58, SEM = 0.69) and attend visual (M = −2.65, SEM = 0.55) conditions (both ts(15) > 4.00, ps < .01). There was no effect of attention condition on the target N1 latency (F(2, 30) = 2.64, p > .09, η2 = 0.15).

Auditory Nontarget Evoked Responses

The effect of attention on N1 amplitude elicited by targets was mirrored in the analysis of the N1 amplitude to nontarget stimuli (F(2, 30) = 4.28, p < .03, η2 = 0.22). Direct comparisons revealed that the attend auditory condition (M = −3.64, SEM = 0.47) had significantly larger N1 amplitude compared with the attend visual condition (M = −2.84, SEM = 0.45; t(15) > 2.67, p < .02). There was a trend for a larger N1 in the attend visual condition compared with the dual task (M = −3.07, SEM = 0.40; t(15) = 1.86, p > .08). There was a significant effect of attention on nontarget N1 latency (F(2, 30) = 4.13, p < .03, η2 = 0.22). Direct comparisons indicated a significantly later N1 in the attend visual condition (M = 138.40, SEM = 2.77) compared with the attend auditory condition (M = 145.61, SEM = 3.25; t(15) = 2.19, p < .05). There was a trend toward a later N1 peak latency in the attend visual compared with the dual attention condition (M = 145.00, SEM = 3.09; t(15) = 2.04, p > .06).

Measuring Late Stages of Information Processing

To understand the effects of continuous dual-task performance on the later stages of information processing, we isolated the P3 ERP component for visual and auditory stimuli. A summary of these results is presented in Table 1.

Visual Target Evoked Responses

Figure 4C shows the average P3 ERP waveforms. There was a significant effect of attention on the visual P3 elicited by target face images (F(2, 30) = 25.35, p < .01, η2 = 0.63). Direct comparisons revealed that this effect was driven by a reduced P3 amplitude in the attend auditory condition (M = 0.09, SEM = 0.38) in comparison with the attend visual (M = 4.66, SEM = 0.66) and dual attention (M = 3.97, SEM = 0.66) conditions (both ts(15) > 5.37, ps < .01). There was no significant attention effect on P3 latency (F(2, 30) = 1.74, p > .19, η2 = 0.10).

Visual Nontarget Evoked Responses

There was no effect of attention on P3 amplitude elicited by nontarget stimuli (F(2, 30) = 0.82, p > .45, η2 = 0.05). There was also no effect of attention on the P3 latency to nontarget stimuli (F(2, 30) = 0.72, p > .49, η2 = 0.05).

Auditory Target Evoked Responses

The grand-averaged ERPs for the auditory P3 component are shown in Figure 4D. The auditory-evoked P3 amplitude to targets showed a significant effect of attention (F(2, 30) = 16.34, p < .01, η2 = 0.52). Direct comparisons indicated that the P3 in the attend auditory condition (M = 4.98, SEM = 1.04) and the dual-task condition (M = 3.41, SEM = 0.60) were both larger than in the attend visual condition (M = −1.27, SEM = 0.95; both ts(15) > 4.46, ps < .01), although the attend auditory and dual-task conditions were not significantly different from each other (t(15) = 1.48, p > .15). For P3 latency to auditory targets, there was a significant effect of attention (F(2, 30) = 8.16, p < .01, η2 = 0.35). Direct comparisons revealed that the P3 in the dual-task condition (M = 582.98, SEM = 23.13) peaked significantly later than in both the attend auditory (M = 555.52, SEM = 39.51) and attend visual (M = 517.55, SEM = 72.62) conditions (both ts(15) > 2.73, ps < .02). Furthermore, the peak in the attend visual condition was earlier than in the attend auditory condition (t(15) = 2.14, p < .05).

Auditory Nontarget Evoked Responses

There was no effect of attention condition for nontarget P3 amplitude (F(2, 30) = 0.60, p > .55, η2 = 0.04). There was also no effect of attention for P3 latency to nontargets (F(2, 30) = 0.79, p > .46, η2 = 0.05).

Measuring the Relationship between Behavioral Performance and Neural Activity

To investigate the relationship between neural activity and behavior, we performed several rank order analyses. These analyses were done separately using differences in ERP amplitude and ERP latency from single to dual attention conditions (i.e., attend dual − attend visual/auditory).

Amplitude Differences

Even in the absence of dual attention differences at the group level, such as with the visual P1 amplitude, it is plausible that individual variability in the ERP amplitudes would be effective at rank-ordering individuals in terms of their behavioral performance. The drop-off in amplitude of the visual P1 to targets was not effective at rank ordering individuals in terms of their behavioral performance measured with either d′ (M = 0.46, SEM = 0.03) or RT (M = 0.47, SEM = 0.03; both ts(30) < 1.08, ps > .10, qs > 0.20). However, the reduction in the amplitude of the N1 in the dual attention condition relative to the attend auditory condition could successfully rank order the corresponding dual-task changes in d′ (M = 0.70, SEM = 0.03; t(30) = 7.89, p < .01, q < 0.01) but not the changes in RT (M = 0.51, SEM = 0.03; t(30) = 0.35, p > .73, q > 0.73). These results indicate that the reductions in performance in the dual-task condition relative to the single-task condition measured with d′ are associated with attenuated early sensory processing measured by the auditory N1 evoked by targets.

For the visual-evoked P3, reductions in amplitude were significantly related to both declines in visual task d′ (M = 0.68, SEM = 0.03) and delays in RT (M = 0.63, SEM = 0.02; both ts(30) > 5.90, ps < .01, qs < 0.01). Changes in the auditory P3 were similarly related to delays in RT (M = 0.58, SEM = 0.03; t(30) = 2.93, p < .01, q < 0.02). However, in contrast to the visual P3, reductions in auditory P3 amplitude were negatively related to d′ performance (M = 0.43, SEM = 0.03; t(30) = 2.60, p < .02, q < 0.03). Consequently, it appears that reductions in the visual P3 amplitude were related to d′ and RT decrements, whereas reductions in auditory P3 amplitude were related to improved d′ but slower RT. Figure 5A shows the results of these amplitude rank order analyses.

Figure 5. 

Rank order analysis using the amplitude difference (attend visual/auditory − dual attention) to order the behavioral outcome changes of participant pairs. (A) The average rank order accuracy using amplitude measures from the visual P1, auditory N1, visual P3, and auditory P3 ERP components. (B) The average rank order accuracy using latency measures from the visual P1, auditory N1, visual P3, and auditory P3 ERP components. Blue bars represent the rank order accuracy of changes in sensitivity d′ measures, and red bars represent the rank order accuracy of changes in RT measures. Error bars represent SEM calculated using a jackknifing method. The dashed lines reflect the 95% confidence interval for a random permutation test. Single asterisks (*) represent significance at p < .05, and double asterisks (**) represent significance at p < .01.

Figure 5. 

Rank order analysis using the amplitude difference (attend visual/auditory − dual attention) to order the behavioral outcome changes of participant pairs. (A) The average rank order accuracy using amplitude measures from the visual P1, auditory N1, visual P3, and auditory P3 ERP components. (B) The average rank order accuracy using latency measures from the visual P1, auditory N1, visual P3, and auditory P3 ERP components. Blue bars represent the rank order accuracy of changes in sensitivity d′ measures, and red bars represent the rank order accuracy of changes in RT measures. Error bars represent SEM calculated using a jackknifing method. The dashed lines reflect the 95% confidence interval for a random permutation test. Single asterisks (*) represent significance at p < .05, and double asterisks (**) represent significance at p < .01.

Latency Differences

Although there were no differences between single and dual attention conditions in latency for visual P1 or auditory N1 components elicited by targets, individual differences in latency may still be related to differences in behavior. The change in latency of the visual P1 did significantly rank order changes in d′ above chance (M = 0.56, SEM = 0.02; t(30) = 2.48, p < .02, q < 0.04) but did not significantly rank order changes in RT (M = 0.53, SEM = 0.02; t(30) = 1.55, p > .13, q > 0.17). Differences in the auditory N1 latency did not significantly rank order d′ (M = 0.49, SEM = 0.03; t(30) = 0.41, p > .68, q > 0.68) but did significantly rank order RT differences (M = 0.73, SEM = 0.02; t(30) = 9.75, p < .01, q < 0.01). In other words, delayed visual P1 latencies from single to dual attention conditions were related to decreases in d′, whereas delayed auditory N1 latencies were related to delayed RTs.

Changes in the latency of the P3 component from the visual task were not significantly related to changes in either d′ (M = 0.50, SEM = 0.02) or RT (M = 0.46, SEM = 0.02; both ts(30) < 1.65, ps > .11, qs > 0.15). Changes in the auditory P3 latency were significantly related to both decreases in d′ (M = 0.62, SEM = 0.02) as well as delays in RT (M = 0.58, SEM = 0.02; both ts(30) > 3.79, ps < .01, qs < 0.01). These results are displayed in Figure 5B.

DISCUSSION

This study was designed to investigate the effects of multisensory continuous dual-task performance on the early and late stages of information processing by measuring visual- and auditory-evoked neural activity using the ERP technique and relating the changes in neural activity to individual differences in behavior. The results revealed that dual-task impairments in information processing were mainly restricted to the auditory task, with both early and late auditory processing stages being affected by the dual attention condition. Furthermore, individual differences in both early and late ERP components were related to individual differences in behavioral performance. These results provide key insights to the neural mechanisms underlying the performance of concurrent continuous tasks in multiple sensory modalities.

Dual-task Modulations of Behavioral Performance

There were two noteworthy patterns of behavioral performance. First, across all conditions, the visual task was performed less accurately than the auditory task. Behavioral responses in the visual task were also faster compared with the auditory task, suggesting a generalized speed–accuracy trade-off. Second, the level of performance on the visual task was unaffected by the simultaneous demands of the auditory task. In contrast, the auditory task showed reduced RT and an unchanged d′ when the visual task was performed concurrently in the dual attention condition. The lack of a dual-task performance decrement on visual task discrimination sensitivity is consistent with studies showing that auditory tasks do not always interfere with visual task performance (Drews et al., 2008; Kunar et al., 2008).

There are two differences between the visual and auditory tasks that may account for the behavioral results. First, the stimulus presentation rate was faster in the visual task than in the auditory task. Second, the stimuli in the visual task were embedded in noise and briefly presented, whereas the auditory stimuli were not degraded. The combination of these differences likely resulted in the auditory task being less difficult than the visual task, a proposal that is supported by the higher mean d′ in the auditory task. The high level of difficulty of the visual task compared with the auditory task, a result of increased presentation rate and degraded stimuli, may have led participants to place priority on this task over the auditory task, although they were instructed to give the tasks equal priority. Indeed, the increased RT in the auditory task during the dual attention condition suggests that participants were sacrificing performance in the easier auditory task to maintain performance levels in the visual task.

Dual-task Modulations of Neural Activity

If participants were prioritizing the visual task because of its difficulty, then the dual attention condition should affect auditory processing but not visual processing. Consistent with this hypothesis, the amplitude and latency of the visual P1 component were the same in the attend visual and attend dual conditions. Although the P1 amplitude was similar between single and dual tasks for both targets and nontargets, visual stimuli in the attend auditory condition elicited a smaller amplitude and later peaking P1 component. This finding is in line with previous studies showing that the P1 is enhanced when attended compared with when ignored (Hillyard, Teder-Salejarvi, & Munte, 1998; Mangun & Hillyard, 1990). Interestingly, despite the lack of group differences between the attend visual and dual attention conditions in terms of the P1 latency to targets, rank order analyses revealed that participants who had more delayed P1 peaks in the dual attention versus attend visual condition also tended to have lower d′ in the dual attention compared with the attend visual condition. This may mean that the concurrent performance of both the visual and auditory tasks delayed early visual processing and the degree of this delay contributed to reductions in the ability to accurately detect targets in the visual task.

Consistent with the hypothesis that performance in the auditory task was sacrificed in favor of the visual task in the dual attention condition, the amplitude of the auditory N1 was smaller in response to target stimuli in the attend dual condition compared with the attend auditory condition. The influence of the dual task on the amplitude of the auditory N1 evoked by targets indicates that early processing of auditory stimuli was disrupted when attention was split between the visual and auditory tasks, a finding consistent with previous studies (e.g., Singhal et al., 2002). Similarly, RT in the auditory task was increased in the dual task compared with the single task. Unlike previous studies, however, we investigated how these changes in ERP amplitudes related to the changes in behavioral performance. The smaller N1 amplitude caused by the dual attention condition was strongly related to the corresponding decrements in d′ for each participant. Furthermore, delays in the N1 peak induced by the dual attention condition were strongly related to delays in RT. These findings suggest that the variations in N1 amplitude and latency to auditory targets under dual attention conditions can be used to accurately index and even predict the level of auditory processing interference in a dual task.

Similar to the P1 amplitude analysis, the amplitude of the visual target evoked P3 was larger in attend visual and dual tasks compared with the attend auditory condition. This is consistent with previous studies showing larger P3 amplitudes when infrequent stimuli are attended and detected (Duncan-Johnson & Donchin, 1977; Sutton et al., 1965). However, other studies have found reduced P3 amplitudes when attention is divided (de Jong et al., 2010; Mangun & Hillyard, 1990), which does not match with the similar P3 amplitudes evoked by visual targets under single- and dual-task conditions observed here. This disparity could be explained by (a) the hypothesized increased priority given to the visual task over the auditory task, (b) the finding that the P3 for a primary task can be unaffected by changes in difficulty of a secondary task (Singhal et al., 2002), (c) the participants engaged in a strategy of switching between tasks because of the fixed presentation rate in the auditory task, or (d) some combination thereof. Nevertheless, the delay in auditory P3 to targets during the dual task is consistent with delays attributed to a processing bottleneck in discrete dual-task studies (Sigman & Dehaene, 2008; Dell'Acqua et al., 2005; Arnell et al., 2004; Luck, 1998). When considered along with the amplitude reduction seen in the dual-task N1 to targets, this provides further support for the idea that participants in this experiment sacrificed auditory task performance to prioritize the visual task under dual attention conditions.

The rank order relationships between changes in behavioral performance and P3 amplitude and latency changes indicate that later neural processing stages are affected by dual-task conditions in a way that is also related to behavior. These results match well with previous findings that P3-related ERP activity elicited by visual stimuli could explain large amounts of variance in RT (Gerson et al., 2005). Although the negative relationship between auditory P3 amplitude and reduced d′ was unexpected, this effect may reflect individual differences in strategic speed–accuracy trade-offs. Because the same auditory P3 amplitude reductions are related to decreases in RT, smaller P3 responses in the dual task appear to have led to slower RT but either no reductions or modest increases in d′. This potential trade-off effect is small compared with the other P3 rank order effects. Moreover, the latency delays in the same auditory P3 components are very strongly related to both decreases in d′ and delays in RT. Similar to the N1, changes in the visual and auditory P3 could be used to index changes in behavior in dual-task environments, such as is currently done with brain–computer interfaces (Cecotti, Kasper, Elliott, Eckstein, & Giesbrecht, 2011; Gevins & Smith, 2003).

Implications and Conclusions

The finding that overall visual processing was unaffected by the simultaneous attention to the auditory word monitoring task supports the idea that auditory processing does not necessitate visual interference (Gherri & Eimer, 2011; Kunar et al., 2008). Instead, previous evidence shows that, whereas simply listening to auditory information may not interfere with visual processing (Drews et al., 2008), producing auditory messages (Kunar et al., 2008) and actively storing auditory messages in memory (Gherri & Eimer, 2011) can cause interference with visual tasks measured by both behavior and neural activity. The present findings further reveal that processing of auditory stimuli, here, single words that must be identified and compared with the known target word, may not be sufficient to interfere with visual processing. On the other hand, the present results seem to clearly indicate visual task interference on auditory information processing at both early and late stages. The precise nature of this interference is unclear based on the present experiment; however, the pattern of results is consistent with the notion that, at some stage of processing, vision and audition may share a single pool of attentional resources in continuous dual-task situations (Arrighi, Lunardi, & Burr, 2011) or that both modalities require access to a single capacity limited mechanism that is subject to information processing bottlenecks (Sigman & Dehaene, 2008; Dell'Acqua et al., 2005; Arnell et al., 2004; Luck, 1998; Pashler, 1994). Importantly, within the present context, the interaction across modalities is likely partially driven by the difficulty of the visual task, which may have served to either draw extra resources that would otherwise have been used for the auditory task or to create an information processing bottleneck.

What is clear from the present data is that the resource sharing or bottleneck responsible for the changes in auditory-evoked responses are first occurring within 200 msec of stimulus presentation, suggesting that the information that is being affected is relatively early in the information processing stream. The N1 has previously been localized to auditory cortex (Woldorff et al., 1993), and the dual-task modulation of this N1 response likely reflects attenuated early sensory processing under dual-task conditions. On the other hand, the P3 modulations around 400–600 msec after stimulus presentation, which is thought to reflect changes in attention at later decision stages in higher-order brain regions (Mangun & Hillyard, 1990; Duncan-Johnson & Donchin, 1977), may be mediated by a separate resource pool or bottleneck. Consistent with this notion, a separate control analysis did not reveal a correlation between dual-task modulations in N1 and P3 amplitude or latency (all r values < 0.17, p > .55). Although we cannot determine whether the observed patterns of data reflect resource sharing or a bottleneck, the present data demonstrate separate dual-task detriments in both early and later auditory neural processing.

It is difficult to fully dissociate the sensory modality and the difficulty of the tasks in this study, and as a result, the differences arising between the visual and auditory tasks cannot be attributed to a single cause without further experimentation. Previous work has shown similar auditory N1 and P3 effects with systematically manipulated difficulty in audiovisual dual tasks (de Jong et al., 2010; Singhal et al., 2002), but these studies used different types of tasks and did not completely characterize the relationship between neural activity and behavior. Future work using continuous performance tasks is needed to relate neural signals from both modalities to behavior while also manipulating the difficulty of the two tasks. Furthermore, additional studies should continue to emphasize the relationships between behavior and properties of the N1 and P3, as these relationships could potentially be used as a reliable index and predictor of detection performance.

In summary, the present results support the conclusion that, during a multisensory continuous performance dual task, auditory processing was reduced by the dual-task demands whereas visual processing was unaffected, a pattern that may have resulted from strategies to mitigate the increased difficulty of the visual task. Furthermore, the results also support the conclusion that the increased attentional demands imposed by a multisensory continuous performance dual task not only modulate neural activity associated with both early and late stages of information processing but that these modulations are tightly linked with behavioral performance.

Acknowledgments

This research was supported by the Institute for Collaborative Biotechnologies through contract W911NF-09-D-0001 from the U.S. Army Research Office.

Reprint requests should be sent to Ryan W. Kasper, Department of Psychological and Brain Sciences, University of California, Santa Barbara, CA 93106, or via e-mail: kasper@psych.ucsb.edu.

REFERENCES

REFERENCES
Allport
,
D. A.
,
Antonis
,
B.
, &
Reynolds
,
P.
(
1972
).
On the division of attention: A disproof of the single channel hypothesis.
Quarterly Journal of Experimental Psychology
,
24
,
225
235
.
Arnell
,
K. M.
,
Helion
,
A. M.
,
Hurdelbrink
,
J. A.
, &
Pasieka
,
B.
(
2004
).
Dissociating sources of dual-task interference using human electrophysiology.
Psychonomic Bulletin & Review
,
11
,
77
83
.
Arrighi
,
R.
,
Lunardi
,
R.
, &
Burr
,
D.
(
2011
).
Vision and audition do not share attentional resources in sustained tasks.
Frontiers in Psychology
,
2
,
56
.
Benjamini
,
Y.
, &
Hochberg
,
Y.
(
1995
).
Controlling the false discovery rate—A practical and powerful approach to multiple testing.
Journal of the Royal Statistical Society, Series B, Methodological
,
57
,
289
300
.
Brisson
,
B.
,
Robitaille
,
N.
, &
Jolicoeur
,
P.
(
2007
).
Stimulus intensity affects the latency but not the amplitude of the N2pc.
NeuroReport
,
18
,
1627
1630
.
Cecotti
,
H.
,
Kasper
,
R. W.
,
Elliott
,
J. C.
,
Eckstein
,
M. P.
, &
Giesbrecht
,
B.
(
2011
).
Multimodal target detection using single trial evoked EEG responses in single and dual-tasks.
Conference Proceedings: IEEE Engineering in Medicine and Biology Society
,
2011
,
6311
6314
.
Das
,
K.
,
Giesbrecht
,
B.
, &
Eckstein
,
M. P.
(
2010
).
Predicting variations of perceptual performance across individuals from neural activity using pattern classifiers.
Neuroimage
,
51
,
1425
1437
.
de Jong
,
R.
(
1993
).
Multiple bottlenecks in overlapping task-performance.
Journal of Experimental Psychology: Human Perception and Performance
,
19
,
965
980
.
de Jong
,
R.
,
Toffanin
,
P.
, &
Harbers
,
M.
(
2010
).
Dynamic crossmodal links revealed by steady-state responses in auditory–visual divided attention.
International Journal of Psychophysiology
,
75
,
3
15
.
Dell'Acqua
,
R.
,
Jolicoeur
,
P.
,
Vespignani
,
F.
, &
Toffanin
,
P.
(
2005
).
Central processing overlap modulates P3 latency.
Experimental Brain Research
,
165
,
54
68
.
Drews
,
F. A.
,
Pasupathi
,
M.
, &
Strayer
,
D. L.
(
2008
).
Passenger and cell phone conversations in simulated driving.
Journal of Experimental Psychology: Applied
,
14
,
392
400
.
Duda
,
R. O.
,
Hart
,
P. E.
, &
Stork
,
D. G.
(
2001
).
Pattern classification
(2nd ed.).
New York
:
Wiley
.
Duncan-Johnson
,
C. C.
, &
Donchin
,
E.
(
1977
).
On quantifying surprise: The variation of event-related potentials with subjective probability.
Psychophysiology
,
14
,
456
467
.
Gerson
,
A. D.
,
Parra
,
L. C.
, &
Sajda
,
P.
(
2005
).
Cortical origins of response time variability during rapid discrimination of visual objects.
Neuroimage
,
28
,
342
353
.
Gevins
,
A. S.
, &
Smith
,
M. E.
(
2003
).
Neurophysiological measures of cognitive workload during human–computer interaction.
Theoretical Issues in Ergonomics Science
,
4
,
113
131
.
Gherri
,
E.
, &
Eimer
,
M.
(
2011
).
Active listening impairs visual perception and selectivity: An ERP study of auditory dual-task costs on visual attention.
Journal of Cognitive Neuroscience
,
23
,
832
844
.
Green
,
D. M.
, &
Swets
,
J. A.
(
1966
).
Signal detection theory and psychophysics.
New York
:
Wiley
.
Hillyard
,
S. A.
,
Hink
,
R. F.
,
Schwent
,
V. L.
, &
Picton
,
T. W.
(
1973
).
Electrical signs of selective attention in the human brain.
Science
,
182
,
177
180
.
Hillyard
,
S. A.
,
Teder-Salejarvi
,
W. A.
, &
Munte
,
T. F.
(
1998
).
Temporal dynamics of early perceptual processing.
Current Opinion in Neurobiology
,
8
,
202
210
.
Isreal
,
J. B.
,
Chesney
,
G. L.
,
Wickens
,
C. D.
, &
Donchin
,
E.
(
1980
).
P300 and tracking difficulty: Evidence for multiple resources in dual-task performance.
Psychophysiology
,
17
,
259
273
.
Jacoby
,
O.
,
Hall
,
S. E.
, &
Mattingley
,
J. B.
(
2012
).
A crossmodal crossover: Opposite effects of visual and auditory perceptual load on steady-state evoked potentials to irrelevant visual stimuli.
Neuroimage
,
61
,
1050
1058
.
Joassin
,
F.
,
Maurage
,
P.
,
Bruyer
,
R.
,
Crommelinck
,
M.
, &
Campanella
,
S.
(
2004
).
When audition alters vision: An event-related potential study of the cross-modal interactions between faces and voices.
Neuroscience Letters
,
369
,
132
137
.
Just
,
M. A.
,
Keller
,
T. A.
, &
Cynkar
,
J.
(
2008
).
A decrease in brain activation associated with driving when listening to someone speak.
Brain Research
,
1205
,
70
80
.
Kammer
,
T.
,
Lehr
,
L.
, &
Kirschfeld
,
K.
(
1999
).
Cortical visual processing is temporally dispersed by luminance in human subjects.
Neuroscience Letters
,
263
,
133
136
.
Kasper
,
R. W.
,
Elliott
,
J. C.
, &
Giesbrecht
,
B.
(
2012
).
Multiple measures of visual attention predict novice motor skill performance when attention is focused externally.
Human Movement Science
,
31
,
1161
1174
.
Kok
,
A.
(
1997
).
Event-related-potential (ERP) reflections of mental resources: A review and synthesis.
Biological Psychology
,
45
,
19
56
.
Kunar
,
M. A.
,
Carter
,
R.
,
Cohen
,
M.
, &
Horowitz
,
T. S.
(
2008
).
Telephone conversation impairs sustained visual attention via a central bottleneck.
Psychonomic Bulletin & Review
,
15
,
1135
1140
.
Kutas
,
M.
,
McCarthy
,
G.
, &
Donchin
,
E.
(
1977
).
Augmenting mental chronometry: The P300 as a measure of stimulus evaluation time.
Science
,
197
,
792
795
.
Lien
,
M. C.
,
Ruthruff
,
E.
, &
Johnston
,
J. C.
(
2006
).
Attentional limitations in doing two tasks at once—The search for exceptions.
Current Directions in Psychological Science
,
15
,
89
93
.
Luck
,
S. J.
(
1998
).
Sources of dual-task interference: Evidence from human electrophysiology.
Psychological Science
,
9
,
223
227
.
Macmillan
,
N. A.
, &
Creelman
,
C. D.
(
1991
).
Detection theory: A user's guide.
New York
:
Cambridge University Press
.
Mangun
,
G. R.
, &
Hillyard
,
S. A.
(
1990
).
Allocation of visual attention to spatial locations: Tradeoff functions for event-related brain potentials and detection performance.
Perception & Psychophysics
,
47
,
532
550
.
Marti
,
S.
,
Sigman
,
M.
, &
Dehaene
,
S.
(
2012
).
A shared cortical bottleneck underlying attentional blink and psychological refractory period.
Neuroimage
,
59
,
2883
2898
.
Meyer
,
D. E.
, &
Kieras
,
D. E.
(
1997
).
A computational theory of executive cognitive processes and multiple-task performance: 1. Basic mechanisms.
Psychological Review
,
104
,
3
65
.
Parasuraman
,
R.
(
1985
).
Event-related brain potentials and intermodal divided attention.
Proceedings of the Human Factors and Ergonomics Society Annual Meeting
,
29
,
971
975
.
Pashler
,
H.
(
1994
).
Dual-task interference in simple tasks: Data and theory.
Psychological Bulletin
,
116
,
220
244
.
Sergent
,
C.
,
Baillet
,
S.
, &
Dehaene
,
S.
(
2005
).
Timing of the brain events underlying access to consciousness during the attentional blink.
Nature Neuroscience
,
8
,
1391
1400
.
Shams
,
L.
,
Kamitani
,
Y.
,
Thompson
,
S.
, &
Shimojo
,
S.
(
2001
).
Sound alters visual evoked potentials in humans.
NeuroReport
,
12
,
3849
3852
.
Shapiro
,
K. L.
,
Raymond
,
J. E.
, &
Arnell
,
K. M.
(
1997
).
The attentional blink.
Trends in Cognitive Sciences
,
1
,
291
296
.
Sigman
,
M.
, &
Dehaene
,
S.
(
2005
).
Parsing a cognitive task: A characterization of the mind's bottleneck.
PLoS Biology
,
3
,
334
349
.
Sigman
,
M.
, &
Dehaene
,
S.
(
2008
).
Brain mechanisms of serial and parallel processing during dual-task performance.
Journal of Neuroscience
,
28
,
7585
7598
.
Singhal
,
A.
,
Doerfling
,
P.
, &
Fowler
,
B.
(
2002
).
Effects of a dual task on the N100–P200 complex and the early and late Nd attention waveforms.
Psychophysiology
,
39
,
236
245
.
Strayer
,
D. L.
,
Drews
,
F. A.
, &
Crouch
,
D. J.
(
2006
).
A comparison of the cell phone driver and the drunk driver.
Human Factors
,
48
,
381
391
.
Strayer
,
D. L.
, &
Johnston
,
W. A.
(
2001
).
Driven to distraction: Dual-task studies of simulated driving and conversing on a cellular telephone.
Psychological Science
,
12
,
462
466
.
Sutton
,
S.
,
Braren
,
M.
,
Zubin
,
J.
, &
John
,
E. R.
(
1965
).
Evoked-potential correlates of stimulus uncertainty.
Science
,
150
,
1187
1188
.
Telford
,
C. W.
(
1931
).
The refractory phase of voluntary and associative responses.
Journal of Experimental Psychology
,
14
,
1
36
.
Troje
,
N. F.
, &
Bülthoff
,
H. H.
(
1996
).
Face recognition under varying poses: The role of texture and shape.
Vision Research
,
36
,
1761
1771
.
Uchiyama
,
Y.
,
Toyoda
,
H.
,
Sakai
,
H.
,
Shin
,
D.
,
Ebe
,
K.
, &
Sadato
,
N.
(
2012
).
Suppression of brain activity related to a car-following task with an auditory task: An fMRI study.
Transportation Research Part F—Traffic Psychology and Behaviour
,
15
,
25
37
.
Vogel
,
E. K.
,
Luck
,
S. J.
, &
Shapiro
,
K. L.
(
1998
).
Electrophysiological evidence for a postperceptual locus of suppression during the attentional blink.
Journal of Experimental Psychology: Human Perception and Performance
,
24
,
1656
1674
.
Welford
,
A. T.
(
1967
).
Single-channel operation in brain.
Acta Psychologica
,
27
,
5
22
.
Wester
,
A. E.
,
Bocker
,
K. B.
,
Volkerts
,
E. R.
,
Verster
,
J. C.
, &
Kenemans
,
J. L.
(
2008
).
Event-related potentials and secondary task performance during simulated driving.
Accident Analysis & Prevention
,
40
,
1
7
.
Woldorff
,
M. G.
,
Gallen
,
C. C.
,
Hampson
,
S. A.
,
Hillyard
,
S. A.
,
Pantev
,
C.
,
Sobel
,
D.
,
et al
(
1993
).
Modulation of early sensory processing in human auditory cortex during auditory selective attention.
Proceedings of the National Academy of Sciences, U.S.A.
,
90
,
8722
8726
.
Zylberberg
,
A.
,
Slezak
,
D. F.
,
Roelfsema
,
P. R.
,
Dehaene
,
S.
, &
Sigman
,
M.
(
2010
).
The brain's router: A cortical network model of serial processing in the primate brain.
PLoS Computational Biology
,
6
,
e1000765
.