Visual attention selectively enhances the neural response to a task-relevant item. But what happens when an item outside the primary focus of attention is also relevant to the task at hand? In a dual-task fMRI experiment, we studied the responses in retinotopically organized visual cortex in such a situation. Observers performed an attention-demanding task in the fovea while another, unmasked stimulus appeared in the visual periphery. With respect to this latter stimulus, observers attempted to perform either a less or a more attentionally demanding task. Both tasks increased the BOLD response to the peripheral stimulus. Behaviorally, however, only the less demanding task was performed well, whereas the demanding task was carried out near chance. What could explain the discrepancy between BOLD response and behavioral performance? A control experiment revealed that the report of the less demanding feature was severely disturbed by a mask. Moreover, the visual attributes queried by the demanding task had a significantly shorter iconic memory persistence. We conclude that, in the dual-task situation, the focus of attention initially remains with the foveal task, but subsequently shifts to the former location of the peripheral stimulus. Such a belated shift to a peripheral iconic memory (futile in one case, informative in the other) would reconcile the similar BOLD response with the disparate behavioral performance. In summary, our results show that an enhanced BOLD response is consistently associated with attentional modulation, but not with behavioral performance.
Not all information in the visual field can be processed at once. Therefore, attention is selectively directed to enhance visual processing (e.g., Pashler, 1998) of potentially relevant information, reflected at the neural level as an enhanced response in visual cortex (e.g., Kastner & Pinsk, 2004; Reynolds & Chelazzi, 2004). The limited capacity of attention poses a problem when information that is present outside the primary focus of spatial attention is also (potentially) relevant. Fortunately, this information is not immediately lost, as its neural representation lingers after stimulus offset (Major & Tank, 2004). At a psychological level, this persistent activity is referred to as iconic memory from which information can be read out by means of a delayed shift of selective attention (Sperling, 1960). Indeed, a recent experiment found that directing selective attention to the content of iconic memory leads to a similar modulation of visual activity as directing selective attention to actual visually present information (Ruff, Kristjànsson, & Driver, 2007).
However, information in iconic memory decays and a boost of visual activity by a belated shift of attention is not guaranteed to retrieve the relevant feature. Indeed, the extent to which visual information depends on attention for its extraction and the time for which it lingers in iconic memory may be inversely related. What constitutes more and less attention-demanding information has been studied extensively by so-called dual-task paradigms (Pastukhov, Fischer, & Braun, 2009; Koch & Tsuchiya, 2007; Pastukhov & Braun, 2007; Braun, Koch, Lee, & Itti, 2001; Braun, 1998; Braun & Julesz, 1998; Braun & Sagi, 1990). In such a task, subjects are trained on a central task, which is then combined with a second, peripheral task. Because both the central and the peripheral stimuli are masked, subjects have to divide their attention between the central and the peripheral tasks and this division is manipulated by instructing them to give more or less priority to one or the other task. Any resulting tradeoff in performance between the central and peripheral tasks indicates how much attentional resources are required to extract the respective features. Whereas two almost identical non-attention-demanding tasks can be performed in parallel (Pastukhov et al., 2009, Experiment 4; Rousselet, Fabre-Thorpe, & Thorpe, 2002), two very different attention-demanding tasks cannot (Pastukhov et al., 2009, Experiment 2).
Peripheral stimuli that can be reported concurrently with an attention-demanding central task (and thus, require few attentional resources) have been found to be as complex as faces and natural scenes (Reddy, Wilken, & Koch, 2004; Rousselet, Macé, & Fabre-Thorpe, 2003; see VanRullen, Reddy, & Koch, 2004 for a review). However, all such attributes are represented explicitly in neuronal activity (Hung, Kreiman, Poggio, & DiCarlo, 2005; Sugase, Yamane, Ueno, & Kawano, 1999; Oram & Perrett, 1992), and thus, need no further integration after feedforward processing. In contrast, high attentional costs arise when “composite” attributes are to be reported, that can be as seemingly simple as the discrimination of rotated “T” versus “L” shapes (Pastukhov et al., 2009; Koch & Tsuchiya, 2007; Li, VanRullen, Koch, & Perona, 2002; Braun et al., 2001; Lee, Koch, & Braun, 1999). Such attributes involve the integration of information from several locations because they are not represented explicitly in the visual processing hierarchy during feedforward processing (Lamme & Roelfsema, 2000). Many theories agree that it is this selective integration of different parts, features, or elements of an object that requires attentional processing (e.g., Duncan, 1984; Treisman & Gelade, 1980).
Therefore, the decay of information in iconic memory might be more detrimental for more attention-demanding attributes. Whereas global, explicitly represented information can be retrieved if only a single neuron coding this attribute is still veridically active, an attribute that is only implicit in the integrated representation of its constituent elements is more susceptible to decay. Indeed, it has been found that iconic memory report is worse for detailed configurational information than for global properties (Von Wright, 1970). Thus, despite a similar boost of visual activity, a shift of attention to an iconic memory trace might suffice to extract a less attention-demanding feature, whereas not enough information is left to reliably report a more attention-demanding feature.
In the present series of experiments, we will compare activity related to processing of less and more attention-demanding relevant features presented either within or outside the primary focus of attention. To this end, we use a dual-task paradigm with an attention-demanding central task to control the primary focus of attention and a more or less attention-demanding peripheral task with an unmasked stimulus. This allows us to compare situations in which a peripheral stimulus (i) decays from iconic memory without being behaviorally relevant, (ii) is behaviorally relevant but appears initially outside the primary focus of attention, so that it may become attended only after some delay, (iii) is exclusively relevant and can potentially be fully attended.
First of all, in Experiment 1, we determine the differential performance levels for two nonmasked peripheral attributes under dual-task conditions. Whereas a global color feature can be reported together with the attention-demanding central task, the configurational orientation of the same stimulus cannot. In Experiment 2, we show that report of the less attention-demanding color feature under dual-task conditions is disturbed by masking, suggesting that successful performance is due to a readout from iconic memory. In Experiment 3, we find that this tentative readout leads to a different modulation of visual activity as when the relevant stimulus is present in the primary attention focus. When presented in the primary focus of attention, the more attention-demanding task leads to a stronger BOLD response although performance is similar as in the less attention-demanding task. But when a relevant feature is presented outside the primary focus of attention, the modulation of the BOLD response is similar for the less and more attention-demanding task although performance differs dramatically. It thus seems that attention always shifts to the iconic memory trace of a relevant stimulus, even though only its coarse (i.e., less attention-demanding) features can be successfully retrieved. Finally, in Experiment 4, we further delineate the time course of the differential decay from iconic memory for less and more attention-demanding features.
Eleven subjects (6 men, mean age = 22.6 years, range = 20–27 years) participated in Experiment 3, two of which also participated in Experiment 1 (2 women, age = 23–26 years), four in Experiment 2 (2 women, mean age = 26.0 years, range = 25–27 years), and seven in Experiment 4 (3 women, mean age = 22.9 years, range = 21–25 years). Informed consent was obtained from all subjects. All were right-handed and had normal or corrected-to-normal visual acuity. The data of one of the subjects had to be discarded due to ceiling performance (>90%) on the central task in the scanner session.
Displays for Experiments 1, 2, and 4 were generated with MATLAB (version 7.4) and Psychophysics Toolbox (version 1.0.42) running on a Power Mac G5 and presented on an LCD Cinema Display on a background with RGB value 125, 125, 125 (luminance 25 cd/m2). The stimuli for the imaging experiment (Experiment 3) were generated off-line with MATLAB and were presented with Presentation software (Neurobehavioral Systems, Albany, CA) on a backprojection screen that subjects viewed through a mirror mounted on the head coil of the scanner.
Stimulus and Procedure: Experiment 1
We used a standard dual-task paradigm with a highly attention-demanding letter-discrimination task in the center (Li et al., 2002; Lee et al., 1999) and one of two peripheral tasks (Figure 1). In different blocks of trials we instructed subjects to devote attention in different proportions to the central and peripheral tasks. This leads to a performance tradeoff function that reveals the attentional demand of the different peripheral tasks.
In the center of the display, five letters (size 0.12° × 0.12°, RGB: 200, 200, 200, luminance 70 cd/m2) were spread randomly on a circle with a radius of 0.73°. The letters were all Ts, all Ls, one T among four Ls, or one L among four Ts. After an individually determined presentation time (250–300 msec; see below), the letters were masked with five Fs at the same locations. The subjects' task was to decide whether all letters were the same or whether one letter was different.
In the periphery of the display, a single stimulus was presented on a random location at 8° from the center of the display for an individually determined presentation time (17–50 msec; see below), not followed by a mask. This peripheral stimulus was a fractal outline, generated by recursive deflection of all sides from a right-angled triangle with a side length of 2.4°. On each trial, the peripheral stimulus randomly assumed one of four hues (greenish, bluish, yellowish, or reddish; luminance ∼30 cd/m2) and one of four orientations (right angle bottom-left, bottom-right, top-left, or top-right). The two peripheral tasks were to report either the color or the orientation of the peripheral stimulus. In either case, chance performance was 25% correct.
Subjects were asked to maintain fixation throughout the experiment. They were aware that doing so was advantageous in performing the peripheral task, as the peripheral stimulus appeared only briefly at an unpredictable location, making eye movements highly unlikely. Each trial started with the presentation of a fixation cross for 500 msec. Thereafter, central and peripheral stimuli appeared, the central stimuli were masked, and a response screen cued the subjects to respond to the current task(s). A high-pitched beep provided individual error feedback to each response.
Subjects were trained on each task separately (central and the two peripheral tasks) until performance was stable near threshold (84% correct). Task difficulty was adjusted for each subject and task to ensure comparable performance. Central-task difficulty was adjusted by varying presentation time. The difficulties of the peripheral color and orientation tasks were adjusted by changing color saturation and fragmentation of the fractal edges, respectively. After training, subjects performed various task combinations. For 5 hours on subsequent days, subjects performed blocks of only the central task (SingleLetter), only the color task (SingleColor), only the orientation task (SingleOrientation), or a combination between the central task and the color (DualColor) or orientation task (DualOrientation). In the dual-task conditions, subjects additionally received one of three priority instructions: give priority to the central task, priority to the peripheral task, or equal priority to both tasks. Instructions were varied per block of 80 trials in counterbalanced order.
Stimulus and Procedure: Experiment 2
The results of Experiment 1 showed that observers successfully performed the peripheral color task together with the central task. However, the peripheral orientation task was performed close to chance level when priority was given to the central task. The central stimuli could only detain attention until they were masked, and thereafter, in the dual-task conditions, attention would be free to shift to any persistent iconic memory (Sperling, 1960) of the peripheral stimulus (which was not masked). To examine whether performance on the peripheral tasks could be due to such a readout from iconic memory, in Experiment 2 we used the same paradigm as before (Figure 1) but now compared mask and no-mask conditions. In the dual-task conditions, subjects were always encouraged to maintain their primary attention focus on the central task by giving it priority.
The central letters were again masked after an individually determined presentation time (180–330 msec). But now the peripheral stimulus was always masked in the single tasks by 10 fractals with random hues and orientations equally spaced on a circle (for 300 msec), one of them in the same position as the target fractal. The individually determined presentation time of the peripheral stimulus ranged between 40 and 70 msec for the color tasks, and 90 and 150 msec for the orientation tasks. In the dual-task conditions, the peripheral stimulus was masked on half of the blocks (of 50 trials). Subjects were trained on the central and the two peripheral tasks separately until performance was stable, and then performed blocks of the different conditions for 4 hours on subsequent days.
Stimulus and Procedure: Experiment 3
The results from Experiment 2 showed that the mask disturbed peripheral performance in the unmasked dual tasks. This suggests that performance is related to the (lack of) success of readout of the peripheral stimulus from iconic memory. On a neural level, a delayed shift of attention to iconic memory would lead to an increase in activity, both for the DualColor and DualOrientation task. However, whereas in the DualColor task this would be related to reliable report, the delayed attentional modulation would not be related to good performance in the DualOrientation task.
fMRI Procedure: Experiment 3
Stimuli and tasks were the same as in Experiment 2, but the peripheral stimulus was never masked. When subjects had reached stable performance levels on all tasks, they practiced for an hour under the same conditions as subsequently in the scanner (see below). Images were collected on a 3-T Trio-Scanner (Siemens, Germany) with an eight-channel head coil (32 slices, TR = 2000 msec, flip angle = 80°, resolution = 3.5 × 3.5 × 4 mm). An IREPI image was also acquired for each subject (3.5 × 3.5 × 4 mm). Before and after the experimental blocks, we ran a localizer scan to map visual areas responding to the central versus the peripheral visual field so as to distinguish areas processing the central letters from areas processing the peripheral stimulus. The localizer stimulus was a dynamic, high-contrast red and green checker pattern, which was presented in five annular regions of progressively larger radius and followed by a blank pause. The fifth annular region (7°–8° eccentricity) included all possible locations of the peripheral stimulus (see below). During one localizer scan, subjects viewed 10 such cycles, each lasting 24 sec, while maintaining fixation.
The peripheral stimulus appeared randomly at one of four locations at 7.5° eccentricity (i.e., the centers of the four quadrants of the display). Trial timing was individually determined based on the training phase [average presentation time was 280 (SD = 30) msec for the central stimuli and 24 (SD = 13) msec for the peripheral stimulus]. As in Experiment 1, the unpredictable location and brief presentation of the peripheral stimulus made eye movements highly unlikely. No error feedback was provided in the scanner in order to avoid activation related to error processing. The five different conditions (3 single tasks and 2 dual tasks) were presented in mini-blocks of 10 trials with 16-sec fixation periods between mini-blocks, such that the sequence of conditions was varied as much as possible (excluding immediate repeat of a condition). Each run comprised 100 trials (5 mini-blocks × 2 repeats) and lasted approximately 6 min. All subjects completed five full runs.
fMRI Data Analyses: Experiment 3
Data were analyzed with SPM5 (Wellcome Centre for Neuroimaging, UCL, London, UK). Preprocessing of the acquired volumes included correction for slice acquisition time, spatial realignment, normalization to a standard EPI template confirming to the MNI, and smoothing with an 8-mm FWHM Gaussian kernel. The five conditions and six movement regressors (derived from the realignment procedure) were used as covariates in a multiple regression analysis. Activity in the dual and single tasks was compared with a conjunction analysis testing the difference between the DualOrientation and SingleLetter conditions (one-sample t test in a random effects analysis) and masking the result with the comparison between the DualColor and SingleLetter condition.
For the specific activity elicited by central and peripheral stimuli, individual functional volumes of interest (VOIs) were determined on the basis of the localizer scans. For each subject, five eccentricities and six movement regressors were used as covariates in a first-level multiple regression analysis. The individual central VOIs were defined as areas more responsive to stimulation at 0°–1° than at 7°–8° eccentricity, and the peripheral VOIs consisted of areas more responsive to 7°–8° than to 0°–1° eccentricity. Included were occipital clusters with more than 50 voxels and p < .05 (FWE). A general difference across subjects between central and peripheral stimulation was tested with one-sample t tests in a random effects analysis. All results of the random effects analyses were corrected for false discovery rates at a p-threshold of .01 and a cluster threshold of 50 voxels.
A first-level multiple regression analysis with the five conditions and six movement regressors was then performed separately for the individual VOIs using MarsBar (Marseille ROI toolbox). The data were tested separately for the color and orientation tasks. Letter-related activity in the central VOIs was compared between the SingleLetter condition, the respective dual task, and the respective peripheral single task. Activity related to the peripheral visual field was compared between the respective peripheral single task, the respective dual task, and the SingleLetter condition.
Stimulus and Procedure: Experiment 4
In Experiment 4, we tested the time course of iconic memory decay for the color and orientation attributes. To this end, after a fixation period of 500 msec, one, four, or eight peripheral stimuli were presented simultaneously for 25 msec, spaced equally at an eccentricity of 7.5°. A fixation point was present throughout the trial and color and orientation were chosen independently for each peripheral stimulus. One peripheral stimulus was cued 10, 250, or 500 msec after stimulus offset by a white line originating at the fixation point. The nine combinations of cue condition and set size were randomly interleaved from trial to trial. In separate blocks of trials, subjects indicated either the color or the orientation of the cued peripheral stimulus and received auditory error feedback.
Central-task performance was also dependent on instruction conditions in combination with the peripheral orientation task, as shown with an ANOVA [x-axis Figure 2B; F(4) = 315.56, p < .001]. Performance significantly declined from the priority-center (95%) to the priority-equal condition (81%; p < .001, Tukey HSD) and further to near chance in the priority-periphery condition (56%; p < .001, Tukey HSD). Instruction condition also significantly influenced peripheral orientation performance [y-axis Figure 2B; F(4) = 48.07, p < .002]. Compared to the priority-periphery condition (89%), performance was reduced in the priority-equal condition (63%; p < .001, Tukey HSD) and was often at chance in the priority-center condition (43%; p < .001, Tukey HSD).
The combination of the central task with the peripheral color task revealed only a modest performance tradeoff: Both tasks together were performed almost as well as either task alone. This comparative lack of dual-task interference implies that the peripheral color task requires few attentional resources. In contrast, the combination of the central task with the peripheral orientation task showed a substantial performance tradeoff: When either task was performed optimally, performance on the other task was reduced to chance level. Thus, the orientation task requires substantial attentional resources.
Figure 3 presents the normalized performance in Experiment 2 separately for the color (A) and orientation (B) conditions. First of all, ANOVAs showed no significant differences in central-task performance, both when combined with the peripheral color and with the peripheral orientation task [x-axes in Figure 3A and B; for both F(6) < .9, ns]. Therefore, any differences on peripheral performance are not due to a tradeoff with central-task performance.
Nevertheless, ANOVAs on the peripheral-task performance showed a significant effect of conditions both for the color [y-axis in Figure 3A; F(6) = 25.93, p < .001] and the orientation tasks [y-axis in Figure 3B; F(6) = 56.14, p < .001]. Post hoc tests revealed that the results in the conditions without a mask replicated those in the priority-center conditions of Experiment 1. Performance was not lower in the DualColor (98%) than in the SingleColor condition (p > .7, Tukey HSD), but decreased significantly in the DualOrientation (45%) as compared to the SingleOrientation condition (p < .001, Tukey HSD).
The mask had an extremely detrimental effect on performance in the color task where performance dropped from 98% to a mere 52% (p < .001, Tukey HSD). But also performance on the DualOrientation task was lower with a mask (37%) than without (45%; p < .04, Tukey HSD). Finally, planned t tests showed that peripheral performance in the DualColor and DualOrientation conditions differed significantly, both with a mask [52% vs. 37%; t(31) = 3.99, p < .001] and without [98% vs. 45%; t(33) = 17.46, p < .001].
The decrease in performance in the DualColor condition with a mask is consistent with the idea that reliable report of color under dual-task conditions without a mask is, at least partially, due to a delayed readout of information from iconic memory. The difference in peripheral performance in the conditions with a mask is consistent with previous dual-task experiments. It suggests that, under conditions of divided attention, a global color feature can be extracted with fewer attentional resources than composite orientation information.
Behavioral performance during the imaging sessions is summarized in Figure 4. Averaged across subjects, central-task performance decreased from 79% correct under single-task conditions to 71% correct in combination with the peripheral color [χ2(1) = 13.82, p < .001] and orientation task [χ2(1) = 16.89, p < .001]. The difference in central-task performance between the two dual tasks was not significant [χ2(1) < 0.20], suggesting that comparable attentional resources were allocated to the center in both conditions.
The difficulty of the peripheral tasks was adjusted successfully, as there was no significant difference between the single-task performance values [87% for color, 84% for orientation; χ2(1) = 2.68, p > .1]. Dual-task performance was significantly lower than single-task performance for both peripheral tasks [color: 75% correct, χ2(1) = 41.89, p < .001; orientation: 36% correct, χ2(1) = 481.41, p < .001]. Importantly, dual-task performance was significantly better for the peripheral color than for the peripheral orientation task [χ2(1) = 312.61, p < .001], although attention was equally allocated to the central stimuli in both conditions, consistent with the disparate attentional demands of the two peripheral tasks established in Experiment 1.
SPM Analysis of Single- versus Dual-task Activation
We first compared activation under dual-task and single-task conditions (Figure 5A). Consistent with earlier studies that implicate a right fronto-parietal network in attentional control (e.g., Giesbrecht, Woldorff, Song, & Mangun, 2003; Jovicich et al., 2001), the BOLD response in right medial frontal gyrus (BA 9; Talairach: 44, 26, 32) was significantly larger under dual-task conditions [t(10) = 12.40, p < .005]. In addition, the dual-task BOLD response was larger in parietal precuneus [BA 7; −6, −64, 54; t(10) = 14.46, p < .005] and in right inferior parietal lobule [BA 40; 42, −40, 52; t(10) = 12.02, p < .005]. This enhanced activity in frontal and parietal areas may reflect the additional working memory load and/or the increased coordination of attentional resources required by two concurrent tasks (Reddy, Moradi, & Koch, 2007; D'Esposito et al., 1995).
SPM Analysis of the Localizer Runs
We then performed a random effects analysis of the localizer scans, in order to identify areas across subjects that were more responsive to central (0°–1° eccentricity) and to peripheral (7°–8° eccentricity) visual stimuli, respectively. For the most central compared to the most eccentric localizer condition, activity was higher in large and approximately symmetric areas left [BA 19; −34, −90, 2; t = 21.71, p < .001] and right [BA 18; 26, −96, 0; t = 16.61, p < .001] of the central pole (Figure 5B). Activity was higher for the most eccentric than for the most central localizer condition in medial areas centered on lingual gyrus (BA 18) and cuneus (BA 17), with peak activity located at 10, −74, −6 (t = 18.25, p < .001) (Figure 5C).
Analyses of Individual Central and Peripheral Activity
Next, we compared the activity in central and peripheral VOIs at an individual level for the different task conditions. We were particularly interested in the activities associated, respectively, with display regions outside the primary focus of attention but, nevertheless, relevant for report and with display regions being in the primary focus of attention (see insets in Figure 6 for a schematic depiction). To identify activity associated with a relevant but not primarily attended display region, we compared a dual-task situation with a single-task-elsewhere situation. For example, the peripheral VOI contrast “DualColor–SingleLetter” reveals the activity difference between the peripheral stimulus being relevant but not primarily attended and the same stimulus being irrelevant. To obtain activity related to exclusive relevance, we compared single-task situations concerning different display regions. For example, the central VOI contrast “SingleLetter–SingleColor” reveals the activity difference between central stimuli being exclusively relevant and the same stimuli being irrelevant.
Figure 6 shows the difference in contrast values between dual- and single-task conditions for central and peripheral VOIs (horizontal and vertical axes, respectively) and for both combinations of central and peripheral tasks (letter–color and letter–orientation). Considering the central VOIs and the letter–color task combination (Figure 6A, horizontal axis), relevance of the central stimuli raised activity by 1.89 ± 0.26 (mean ± standard error; DualColor–SingleColor), whereas exclusive relevance increased it by 3.36 ± 0.26 (SingleLetter–SingleColor). Similar results were obtained for the letter–orientation task combination (Figure 6B, horizontal axis), where relevance enhanced central activity by 2.89 ± 0.25 (DualOrientation–SingleOrientation) and exclusive relevance raised it by 5.05 ± 0.26 (SingleLetter–SingleOrientation). The further increase in BOLD response in the single- than in the dual-task conditions is probably related to a stronger attentional enhancement in the central VOIs when the letters are the only task-relevant stimuli.
Peripheral activation related to the color task (Figure 6A, vertical axis) increased by 1.75 ± 0.27 when the peripheral feature had to be reported (DualColor–SingleLetter), but did not increase further (1.27 ± 0.29) with exclusive relevance (SingleColor–SingleLetter). The similar increase in peripheral BOLD response in the DualColor and SingleColor conditions co-occurred with a similar peripheral performance level in both tasks. Apparently, BOLD activity associated with the performance of an attentionally undemanding task is not modulated by the primary distribution of attention.
On the other hand, peripheral activation associated with the orientation task (Figure 6B, vertical axis) increased by 1.89 ± 0.28 when the peripheral feature was relevant for report (DualOrientation–SingleLetter) and significantly further (3.10 ± 0.29) when the peripheral stimulus was exclusively relevant (SingleOrientation–SingleLetter). The increase in activity when the stimulus was relevant but not in the primary focus of attention was not accompanied by good behavioral performance. Apparently, report of the peripheral orientation is contingent on the further increase in activity with exclusive relevance when the peripheral stimulus is potentially fully attended.
Finally, we compared BOLD activity associated with the single color and orientation tasks. In the peripheral VOIs, activity was higher by 1.83 ± 0.28 when subjects performed the attention-demanding orientation task (SingleOrientation–SingleColor). In contrast, in the central VOIs, activity was lower by 1.69 ± 0.25 during the orientation task (SingleOrientation–SingleColor). These differences may reflect the disparate attentional demands of the tasks. In the less attentionally demanding color task, more attention is “left over” for the central stimuli than during the more demanding orientation task. This account assumes the mandatory allocation of leftover attention to behaviorally irrelevant stimuli, an assumption supported by a substantial body of previous work (Schwartz et al., 2005; Yi, Woodman, Widders, Marois, & Chun, 2004; Rees, Frith, & Lavie, 1997).
We suspected that the differential results for the color and orientation tasks are due to a differential persistence of the color and orientation attributes in iconic memory. Therefore, in Experiment 4, we probed iconic memory by presenting several (1, 4, or 8) peripheral stimuli simultaneously and, at varying times after stimulus offset, cued observers to report on one particular stimulus. The cue delays used were 10, 250, and 500 msec. Figure 7 shows the behavioral performance for the color and orientation conditions at the different cue delays and set sizes. For Set Size 1, performance on either the color or orientation task did not depend significantly on cue delay, suggesting that working memory capacity for the peripheral tasks was not challenged at this set size. At Set Size 4, orientation performance but not color performance declined significantly with cue delay [χ2(2) = 14.00, p < .001]. At Set Size 8, the performance of both tasks declined with cue delay [color: χ2(3) = 33.42, p < .001; orientation: χ2(3) = 182.65, p < .001].
The difference between color and orientation performance was already significant for Set Size 4 at a cue delay of 10 msec [86.5% vs. 77.4% correct, respectively; χ2(1) = 7.56, p < .006], but the largest performance differences between the color and orientation tasks were observed for Set Size 8 and cue delays of 250 msec [73.3% vs. 51.7%; χ2(1) = 28.33, p < .001] and 500 msec [69.4% vs. 43.8%; χ2(1) = 35.25, p < .001]. In the color task, performance at Set Size 4 only differed significantly from Set Size 1 at a delay of 500 msec [82.5% vs. 91.6%; χ2(1) = 10.44, p = .001], whereas in the orientation task this was already the case at a delay of 10 msec [77.4% vs. 94.8 at Set Sizes 4 and 1; χ2(1) = 35.28, p < .001]. Performance on both tasks differed significantly between Set Size 8 and Set Size 1 for all delays [χ2(1) > 6.70, p < .01].
In three psychophysical experiments and one functional imaging experiment, we examined the neural processing of relevant items presented outside the primary focus of attention. In Experiment 1, we determined that the color task could be performed together with the attention-demanding central task, whereas the orientation task could not. The results of Experiment 2 showed that color performance in the dual task was disturbed by masking, suggesting that successful report of a nonmasked peripheral stimulus relies on readout from iconic memory. The imaging data from Experiment 3 showed an equal increase in BOLD response when the peripheral target was relevant to either the color or the orientation task but presented outside the primary focus of attention, consistent with the idea of a belated shift of the attention focus to an iconic memory representation of the peripheral target, which is futile in the case of the attention-demanding task. A final experiment delineated the differential decay of the less and more attention-demanding attributes.
Neural Responses to Central Targets
In our imaging experiment, we compared the neural correlates of visually successful or unsuccessful processing under single- and dual-task conditions. In areas of visual cortex that were responsive to the central visual field, neural activity was least when central letter targets were task-irrelevant, increased when letters were attended under dual-task conditions, and grew even higher when letter targets were attended under single-task conditions. The further increase in the latter situation may reflect a more exclusive focusing of attention in the single-task situation.
Furthermore, central activity was lower when the letters were irrelevant in the single orientation, than in the single color task. Several fMRI studies show that activity related to irrelevant background scenes is inversely related to the difficulty of the foveal task (Schwartz et al., 2005; Yi et al., 2004; Rees et al., 1997), suggesting that even task-irrelevant stimuli may be selected by attention under conditions of low load (see Lavie, 2006 for a review). Presumably, the attentionally undemanding color task allowed observers to allocate some attention to the (task-irrelevant) central targets. This “leftover” attention would have increased neural responses to the behaviorally irrelevant central stimuli, resulting in a higher activity.
Neural Responses to Peripheral Targets
In areas responsive to the peripheral visual field, neural activity was least when peripheral targets were task-irrelevant and increased when these targets were attended under single-task conditions. Although behavioral performance was comparable, the increase was larger for the orientation than for the color task. This difference may reflect the degree to which the behavioral task engages attention (Lavie, 2006): Reporting more attention-demanding features leads to more visual processing and stronger neural activity than reporting less attention-demanding features.
Under dual-task conditions, which forced observers to devote attention primarily to the central targets, neural activity was increased to an equal extent for both the color and orientation tasks. This result may seem quite puzzling, as the behavioral performance could not have been more different: near optimal for the color task and near chance for the orientation task. How could this stark dissociation between neural activity and behavioral performance be explained?
One possibility is that a preparatory signal or baseline shift in anticipation of a task-relevant stimulus increases the neural response outside, and presumably independently, of the focus of attention (e.g., Ress, Backus, & Heeger, 2000; Kastner, Pinsk, De Weerd, Desimone, & Ungerleider, 1999; Shulman et al., 1999; reviewed by Kanwisher & Wojciulik, 2000). Such a preparatory signal would be consistent with a recent study (Reddy et al., 2007) that reported equally enhanced BOLD activity when an attentionally undemanding recognition task was carried out alone, and when it was carried out in combination with an attention-demanding central task. The similar neural modulation combined with the disparate behavioral performance for DualColor and DualOrientation in our experiment could be explained if such a preparatory modulation does not, by itself, ensure perceptual success for attention-demanding stimuli.
However, unlike in the study of Reddy et al. (2007), our peripheral stimulus was not masked and, therefore, the results could also be explained by a belated shift of attention to the former location of the peripheral stimulus. An iconic memory (Sperling, 1960) of this stimulus might remain until the central task has been completed and attention becomes available once again. Depending on the degree to which this memory remains informative, such a belated attention shift may or may not contribute to behavioral performance. Accordingly, a belated and, in some cases, futile attention shift would provide another way in which neural activity could become dissociated from behavioral performance.
Delayed Attention Shift to Iconic Memory
The finding that color report under dual-task conditions was disturbed by the presence of a mask in Experiment 2 strongly supports the latter alternative; good color performance in the dual-task condition without a mask is apparently contingent on information in iconic memory. The difference in dual-task performance between color and orientation report was further explored in Experiment 4 that measured the persistence of an iconic memory for the color and orientation attributes of peripheral targets. Whereas color information was robustly retained for up to 250 msec, orientation information decayed rapidly, consistent with previous reports that more detailed information quickly decays, whereas more global information remains available for later selection (Von Wright, 1970). This can be explained on a neural level because the color of our fractal can, in principle, be represented by a single cell (Dow, 2002). Therefore, it can be retrieved from iconic memory if only one neuron is still veridically active. The orientation of the fractal, on the other hand, is only implicit in the combined representation of its edge elements. One expects, and our results confirm, that this composite attribute is more susceptible to decay.
Accordingly, we propose that the peripheral stimulus appears and disappears while attention is directed at the central task, but that an iconic memory of the peripheral stimulus remains until attention is available once more. At this point, attention shifts to the former location of the peripheral stimulus and attempts to extract the remaining information. This shift to the periphery accounts for the spatially specific enhancement of the BOLD response. In the case of the color task, the relevant information remains available and can be reported reliably. In the orientation task, the relevant information has decayed and performance is at chance.
The hypothesis of a belated attention shift is consistent with the increased activity in a right fronto-parietal network under dual-task conditions. Especially activity in right frontal gyrus has been implicated in the orienting of attention to internal representations (Nobre et al., 2004; see Lepsien & Nobre, 2007 for a review). More specifically, in a recent experiment that contrasted the direction of selective attention to stimuli in the visual field with the readout of information from iconic memory, a similar region was found to be selectively activated in the latter condition (Ruff et al., 2007).
Our results show that the modulation of neural responses closely reflects the allocation of attention. First of all, neural responses to the central visual field increased when attention could be exclusively focused on the central stimuli, relative to the conditions where the peripheral target was also relevant. This was also true for neural activity to the peripheral visual field when the peripheral target could be selectively attended in the attention-demanding task. Furthermore, the neural response to the irrelevant central stimuli was higher in case of the less demanding peripheral task, presumably reflecting the mandatory allocation of leftover attention to irrelevant stimuli (cf. Lavie, 2006).
In contrast, the modulation of neural responses is easily dissociated from behavioral performance. In the dual-task conditions, neural activity to peripheral targets increased to an equal extent for both the less and more demanding task despite a dramatic difference in behavioral performance. Vice versa, when the peripheral stimulus could be exclusively attended, the neural response was much higher for the more demanding than for the less demanding task, although behavioral performance was comparable. It follows that attentional allocation and behavioral performance do not go hand in hand: Not only does attention “spill over” to targets that are irrelevant and unreported, it is even devoted to fading iconic memories that no longer yield perceptually useful information.
We thank Alexandra Grychta for the training of the subjects, Jochem Rieger for providing the localizer procedure, and Eva Deutschmann for developing the deflection-algorithm used to create the coiled-edged triangles. This work was supported by the BMBF Bernstein Network for Computational Neuroscience.
Reprint requests should be sent to Roos Houtkamp, Otto-von-Guericke Universitaet, Department of Cognitive Biology, Leipzigerstrasse 44, Haus 91, 39120 Magdeburg, Germany, or via e-mail: Roos.Houtkamp@gmail.com.