Mental representations of target features (attentional templates) control the selection of candidate target objects in visual search. The question where templates are maintained remains controversial. We employed the N2pc component as an electrophysiological marker of template-guided target selection to investigate whether and under which conditions templates are held in visual working memory (vWM). In two experiments, participants memorized one or four shapes (low vs. high vWM load) before either being tested on their memory or performing a visual search task. When targets were defined by one of two possible colors (e.g., red or green), target N2pcs were delayed with high vWM load. This suggests that the maintenance of multiple shapes in vWM interfered with the activation of color-specific search templates, supporting the hypothesis that these templates are held in vWM. This was the case despite participants always searching for the same two target colors. In contrast, the speed of target selection in a task where a single target color remained relevant throughout was unaffected by concurrent load, indicating that a constant search template for a single feature may be maintained outside vWM in a different store. In addition, early visual N1 components to search and memory test displays were attenuated under high load, suggesting a competition between external and internal attention. The size of this attenuation predicted individual vWM performance. These results provide new electrophysiological evidence for impairment of top–down attentional control mechanisms by high vWM load, demonstrating that vWM is involved in the guidance of attentional target selection during search.
Visual search for known target objects is controlled by representations of the features of these targets (e.g., their color, shape, or size). These representations are described as attentional control settings (e.g., Folk, Remington, & Johnston, 1992) or attentional templates (e.g., Duncan & Humphreys, 1992). Once an attentional template is activated, objects with template-matching features attract attention, whereas template-nonmatching objects do not (e.g., Eimer & Kiss, 2008; Folk & Remington, 1998; Folk et al., 1992). During visual search, active search templates ensure that attention is more likely and more rapidly allocated to possible target objects, thereby improving search efficiency (e.g., Wolfe, 2007). Although the important role of attentional templates for the guidance of visual search is generally acknowledged, the question where these representations are maintained remains the subject of considerable debate. It is often assumed that search templates are held in visual working memory (vWM; see Olivers, Peters, Houtkamp, & Roelfsema, 2011, for a review). However, others (e.g., Carlisle, Arita, Pardo, & Woodman, 2011) have claimed that, in many search tasks, these templates are only kept in vWM for a brief period before they are transferred to a different longer-term memory store.
Evidence for the storage of attentional templates in vWM comes from experiments that used combined vWM and search tasks. The maintenance of additional information in vWM was found to affect performance in a visual search task during the memory maintenance period. For example, responses to target objects in a search display were faster when these objects matched a feature (e.g., a particular color) that was currently held in vWM and slower when a distractor matched this feature, relative to search displays without any memory-matching object (e.g., Soto, Hodsoll, Rotshtein, & Humphreys, 2008; Olivers, Meijer, & Theeuwes, 2006; Downing & Dodds, 2004). This suggests that, when particular features are maintained in vWM, perceptual objects that match these features can attract attention. In other words, vWM representations can act as attentional templates even when they are irrelevant for a currently performed selection task (see Olivers et al., 2011, for further discussion). Such observations provide initial if somewhat indirect support for the hypothesis that the attentional templates that are activated during the preparation for search are also stored in vWM.
If this was the case, the storage of search-unrelated objects in vWM should generally impair search performance, because these objects will interfere with the currently active search template in the same working memory (WM) store. Previous work manipulating verbal WM load has shown that high load impairs the ability to ignore irrelevant distractors (Lavie & De Fockert, 2005), which may also suggest that WM load generally compromises the attentional guidance by target templates (see Lavie, 2005, for a review). However, these studies did not investigate specifically whether loading vWM interferes with visual search performance. When this prediction was tested directly, only limited evidence for load-related interference effects was found. Woodman, Vogel, and Luck (2001) reported that maintaining two or four color objects in vWM produced a constant delay of RTs in a concurrent search task with shape-defined targets but had no effect on search efficiency (measured as the slope of the function linking RTs to the number of items in a search display). These observations, which were confirmed by Oh and Kim (2004, Experiment 2), cast considerable doubt on the hypothesis that search templates are always held in vWM.
According to Carlisle et al. (2011), target templates are only briefly held in vWM and are rapidly transferred to long-term memory in search tasks where target-defining features remain constant, and the same templates can therefore be utilized for an extended period. This was the case in the studies by Woodman et al. (2001) and Oh and Kim (2004), where participants searched for the same target objects throughout the entire experiment. If templates are no longer held in vWM under these conditions, increasing vWM load should have no adverse effects on template-guided search performance, as was indeed observed. Carlisle et al. (2011) obtained more direct evidence for a transfer of search templates from vWM to long-term memory in tasks with constant targets in ERP experiments that recorded the contralateral delay activity (CDA; Vogel & Machizawa, 2004) as an electrophysiological marker of vWM maintenance. They measured CDA components during the preparation for an upcoming search episode across runs of trials where the target-defining feature (a specific color) remained constant. A CDA was present for the first few trials of each run, implying the activation of a preparatory color-specific target template in vWM. However, the CDA disappeared on later trials of the same run, suggesting that this template was no longer held in vWM but had been transferred to a different long-term store.
If the absence of vWM load effects on search performance (Oh & Kim, 2004; Woodman et al., 2001) was the result of such a transfer, such effects should be found under conditions where the identity of a search target changes from trial to trial, and attentional target templates need to be maintained transiently in vWM. This was exactly what was observed in a behavioral study by Woodman, Luck, and Schall (2007). When a particular target shape was cued anew at the start of each trial, a concurrent vWM task impaired search efficiency. In contrast, no such interference effect was obtained when target identity remained constant across all trials. A somewhat different picture emerged in experiments that investigated the impact of WM for locations on template-guided search performance (Oh & Kim, 2004, Experiment 1; Woodman & Luck, 2004). Here, increasing spatial WM load impaired search efficiency even in tasks with constant search targets. However, this does not necessarily reflect an interference between spatial WM load and search templates in WM. The mechanisms involved in maintaining locations in WM are likely to overlap with the mechanisms required for controlling spatial attention during visual search (e.g., Awh, Jonides, & Reuter-Lorenz, 1998). Effects of spatial WM load on search performance may therefore reflect general load-induced impairments in the control of attention shifts during the visual exploration of search displays (Woodman & Luck, 2004), rather than a reduced ability of target templates to guide attentional selectivity.
Overall, these results suggest that search templates for target-defining features are only maintained in vWM when the identity of these target features changes frequently. In the more commonly investigated case where target features remain constant for an extended period, these templates are held in a different long-term memory store. One goal of this study was to challenge the generality of this conclusion. More specifically, we compared the effects of search-unrelated vWM load on the attentional selection of search targets in a task where observers always searched for one specific feature-defined target (e.g., a particular color) and in a more demanding search task where targets were defined by one of two possible colors. Although it has been claimed that constant target templates are always held in a long-term memory store and are therefore not affected by concurrent vWM load (e.g., Woodman et al., 2007), this may not apply to search tasks where several target-defining attributes have to be maintained simultaneously. Such multiple-feature templates appear to be kept in vWM (as reflected by reliable CDA components) even when they remain unchanged for extended periods (Grubert, Carlisle, & Eimer, 2016). If this was the case, template-guided target selection processes should be impaired with high as compared with low concurrent vWM load.
A second goal of this study was to identify the locus of any vWM load-induced impairments of template-guided target selection processes during visual search. Such impairments can affect different stages of visual processing. At an early stage, active attentional templates produce rapid attentional biases toward objects with template-matching features (e.g., Eimer & Kiss, 2008). At a later object identification stage, stored target templates are compared with perceptual objects to determine their status as a target or distractor (see Cunningham & Wolfe, 2014, for a model of visual search that includes an early template-guided attentional selection stage as well as a later memory comparison stage). Because search-unrelated vWM load could interfere with the operation of attentional templates at either or both of these stages, it is difficult to dissociate these two possible loci of load-induced interference effects with behavioral measures alone. Here, we measured N2pc components to target objects in visual search displays as ERP markers of attentional target selection to investigate when and under which conditions the attentional processing of search displays is affected by the load of a concurrent search-unrelated vWM task. The N2pc is an enhanced negativity elicited at posterior scalp electrode sites contralateral to a candidate target object in visual search displays. Typically emerging at around 200 msec poststimulus onset, this component is believed to be generated in ventral extrastriate visual areas (Hopf et al., 2000) and to reflect the emergence of an attentional bias for objects with template-matching features at a relatively early stage of visual processing (e.g., Eimer & Kiss, 2008; Leblanc, Prime, & Jolicoeur, 2008; Lien, Ruthruff, Goodin, & Remington, 2008; Eimer, 1996; Luck & Hillyard, 1994). If the load of a concurrent vWM task affects the operation of attentional biases at such early stages, this should be reflected by systematic differences of N2pc components to search targets under high versus low vWM load.
In the present experiments, a memory sample display that contained either one or four different-shaped objects were presented at the start of each trial. Participants had to maintain these objects during a retention period for a subsequent memory test. Given that average vWM capacity is estimated to be between three and four items (e.g., Cowan, 2010), a memory load of one versus four shapes represents low versus high vWM load, respectively. On some trials, a memory test display was presented immediately after the retention period. This display contained a single object at fixation that either matched or did not match an object in the memory display. On other trials, a search display was presented after the retention interval that included a color-defined target and another distractor object in a nontarget color on opposite sides (see Figure 1). Because no memory test displays were shown on these trials, the information from the sample display that had been maintained during the retention period could be discarded once a search display was presented. However, because attentional templates are activated during the preparation for search, vWM load in the retention interval should still affect the template-guided attentional processing of the search display if these templates are held in vWM. By presenting either a search display or a memory test display on any given trial rather than presenting both successively, search or memory performance could be assessed at the same point in time, immediately after the retention period (see Downing & Dodds, 2004, for a similar method). Another advantage of this procedure is that trials are shorter, and more trials can be included in one testing session, improving the signal-to-noise ratio for EEG-based measures. We chose to use sparse search displays where a target is accompanied by only a single distractor object as target selection should therefore be straightforward. The presence of load-related effects in such a relatively easy selection task would provide strong support for the hypothesis that concurrent vWM load interferes with the guidance of attention by target templates.
In the current Experiment 1, a two-color search task was employed where targets were rectangular bars that were defined by one of two possible colors (e.g., blue or green). Participants had to find the target bar in each search display and report its orientation (horizontal or vertical). Search displays with either target color were equally likely and randomly intermixed within each block. Importantly, this two-color task set remained constant, as each participant searched for the same two target colors throughout the experiment. The critical question was whether such constant multiple-feature templates would be maintained in vWM or in a different long-term store. In the former case, template-guided target selection processes should be impaired with high as compared with low vWM load. If the ability of attentional templates to trigger rapid attentional biases toward target objects was affected by concurrent search-unrelated vWM load, target N2pc components should be attenuated and/or delayed when observers maintain four objects as compared with just one object during the preceding retention period. The absence of any systematic differences between target N2pc components as a function of high versus low vWM load would either imply that vWM load only interferes with the operation of attentional templates at later object identification stages, but not with their ability to trigger rapid attentional biases, or that these templates were not held in vWM but rather in a different longer-term memory store.
Fourteen individuals (M age = 29 years, SD = 6 years; seven men; one left-handed) took part in Experiment 1. All reported normal or corrected-to-normal vision.
Stimuli and Procedure
The experiment was controlled and executed using E-Prime 2.0 software (Psychology Software Tools, Inc.). Stimuli were presented on a 24-in. BenQ monitor (60 Hz; 1920 × 1080 screen resolution) attached to a SilverStone PC, with participant viewing distance at approximately 90 cm. Manual responses were registered via a standard computer keyboard. All stimuli were presented against a black background, with a gray fixation dot (0.2° × 0.2° of visual angle) continuously present throughout each experimental block. The sequence of trial events is illustrated in Figure 1. Each trial started with a memory sample display (200 msec). This was followed by a retention period of 800 msec, after which either a search display (100-msec duration) or a memory test display (2000-msec duration) was shown. The interval between the offset of a search display and the onset of the memory sample display on the next trial was 2900 msec. When a memory test display was shown, the interval between the offset of this display and the start of the next memory sample display was 1000 msec.
All stimuli in the memory sample and test displays were gray (CIE color coordinates: 0.305/0.325). Memory sample displays included one or four outline shapes (square, circle, triangle, hexagon, gate, heart, or star, each covering an area of 1.46° × 1.46°). In low vWM load blocks, a single shape appeared either above or below fixation at an eccentricity of 1.21° (measured relative to the outer edge of this shape). In high vWM load blocks, four different shapes appeared to the left, right, above, and below fixation, each at an eccentricity from fixation of 1.21°. Memory test displays always contained a single shape at fixation. Search displays contained two rectangular bars in two different colors that were oriented horizontally or vertically (0.45° × 0.89°) and were presented directly left or right of fixation at an eccentricity of 1.59° relative to the center of each bar. The set of possible bar colors included red (0.605/0.322), orange (0.543/0.409), yellow (0.405/0.470), green (0.296/0.604), blue (0.169/0.152), magenta (0.270/0.134), and gray (0.305/0.325). All colors were equiluminant (14 cd/m2). The orientation of both bars in the search display was determined randomly and independently on each trial.
Participants were instructed to memorize the item(s) in the memory sample display. On trials where a memory test display appeared after the retention period, they had to report whether or not the item in the memory test display matched an item in the sample display by pressing the “a” or “s” key with their left hand. Matching and mismatching test displays were presented with equal probability. When a search display appeared after the retention period, participants had to select the target bar and report its orientation (horizontal or vertical) by pressing the “0” or “2” key of the numeric keyboard with their right hand. The target bar on any given trial was defined by having one of two possible colors (e.g., red or blue). These two target colors were selected randomly and independently for each participant, except that gray never served as a target color. For each participant, target colors remained constant throughout the experiment. As a reminder of the two target-defining colors, two small uppercase letters were continuously visible at the top of the computer screen during the experiment (e.g., RB for red/blue). The target color bar appeared with equal probability on the left or right side of the search displays, and the color of the distractor bar on the opposite side was selected randomly from the five remaining nontarget colors.
The experiment consisted of 10 blocks (five blocks with low vWM load and five blocks with high vWM load), each containing 36 trials. Search displays appeared on 24 trials, and memory test displays on 12 trials. Seven participants completed the low vWM load blocks before the high vWM load blocks, and this order was reversed for the other seven participants. The two blocked vWM load conditions were both preceded by a practice block of 12 trials. Participants did not perform an additional articulatory suppression task. Such tasks are often employed in vWM studies to prevent the verbalization of visual objects during their maintenance. However, a recent study (Sense, Morey, Prince, Heathcote, & Morey, 2017) showed that the presence versus absence of articulatory suppression has no effect on performance in visual change detection tasks, demonstrating that participants do not engage in the verbal recoding of memorized visual objects in these tasks.
EEG Recording and Data Analysis
EEG was DC-recorded from 27 scalp electrodes, mounted on an elastic cap at sites Fpz, F7, F8, F3, F4, Fz, FC5, FC6, T7, T8, C3, C4, Cz, CP5, CP6, P9, P10, P7, P8, P3, P4, Pz, PO7, PO8, PO9, PO10, and Oz. A 500-Hz sampling rate with a 40-Hz low-pass filter was applied. Channels were referenced online to a left-earlobe electrode and rereferenced off-line to an average of both earlobes. No other filters were applied after EEG acquisition. Trials with eye blinks (exceeding ±60 μV at Fpz), horizontal eye movements (exceeding ±30 μV in the HEOG channels), and muscle movement (exceeding ±80 μV at all other channels) were removed as artifacts, as were trials with incorrect trial responses. The average general EEG data loss on search display trials due to artifacts and response errors was 11% (SD = 13). Following artifact rejection, ERPs were computed separately for trials where a search display or a memory test display was presented after the retention period. EEG was segmented into epochs from 100 msec before to 500 msec after the onset of search or memory test displays, relative to a 100-msec prestimulus baseline. For both types of displays, averaged ERP waveforms were computed for trials with a target in the left or right visual field, separately for low and high vWM blocks. For search displays, N2pc amplitudes were calculated based on ERP mean amplitudes obtained at lateral posterior electrode sites PO7 and PO8 during the 260- to 360-msec interval after search display onset. To compare target N2pc onset latencies between low and high vWM load blocks, a jackknife-based analysis method was employed (see Miller, Patterson, & Ulrich, 1998, for details). This analysis was based on difference waveforms computed by subtracting ERPs at PO7/PO8 ipsilateral to the target from contralateral ERPs. Fourteen subsamples of grand-averaged difference waves were computed, each excluding a different participant from the original sample. Onset latencies were determined as the point in time within the entire 500-msec interval following search display onset where 50% of the N2pc peak amplitude observed in high vWM load blocks was reached, which corresponds to an absolute threshold of −0.41 μV. These latencies were then compared between conditions via paired t tests, with t values corrected according to the formula described by Miller et al. (1998).
Correct RT and error rates for the visual search task were analyzed based on simple paired-sample t tests, comparing low versus high vWM load performance. There was a numerical trend for RTs to search targets to be slower in high as compared with low vWM load blocks (M = 750 vs. 731 msec), but this difference was not statistically reliable (t(13) = 1.35, p > .20). Similarly, there was no significant difference in error rates in the search task between high and low vWM blocks (M = 1.5% vs. 1.2%; t < 1). As expected, performance in the memory-matching task was significantly modulated by vWM load, with higher error rates in blocks with high as compared with low vWM load (M = 19% vs. 2%; t(13) = 6.62, p < .001).
Figure 2 (top) shows ERPs triggered by search displays at electrodes PO7/PO8 contralateral and ipsilateral to the side of the target object, separately for low and high vWM load blocks. Target N2pcs appear to be delayed and attenuated when concurrent vWM load was high. This can be seen more clearly in the N2pc difference waveforms obtained by subtracting ipsilateral from contralateral ERPs (Figure 2, bottom). Whereas target objects elicited a clear N2pc component that emerged around 250 msec in low vWM load blocks, target N2pcs were delayed and smaller in high vWM load blocks. The presence of a reliable effect of vWM load on the target N2pc onset latencies was confirmed a jackknife-based latency analysis. In blocks with low vWM load, the estimated N2pc onset was 250 msec after search display onset, as compared with 319 msec in high vWM load blocks (tc(13) = 3.59, p < .005).
The ANOVA of N2pc mean amplitudes measured in the 260–360 msec poststimulus time window with the factors vWM load (low, high) and Laterality (ipsilateral, contralateral) obtained a significant main effect of Laterality, F(1, 13) = 18.29, p = .001, ηp2 = .59, that was accompanied by an interaction with vWM load, F(1, 13) = 15.58, p < .005, ηp2 = .55. Follow-up analyses comparing contralateral and ipsilateral amplitudes showed that a significant target N2pc component was present during this measurement window in low vWM load blocks (M diff = −0.91 μV; t(13) = 6.67, p < .001), but not in blocks where vWM load was high (M diff = −0.17 μV; t < 1). This absence of reliable target N2pc components with high concurrent vWM load may be due to the fact that these N2pcs were strongly delayed and only emerged during the late part of this 260–360 msec poststimulus window. For this reason, an additional post hoc ANOVA of mean amplitudes measured within a later time window (320–420 msec poststimulus) was conducted. This analysis again revealed a significant interaction between Laterality and vWM load, F(1, 13) = 5.21, p < .05, ηp2 = .29. However, reliable contralateral negativities were now present not only with low vWM load (M diff = −0.80 μV; t(13) = 6.13, p < .001) but also when vWM load was high (M diff = −0.47 μV; t(13) = 4.31, p = .001).
As can be seen in Figure 2 (top), the early visual N1 component in response to search displays were affected by vWM load, with strongly attenuated N1 amplitudes in high vWM load blocks. These load-induced N1 amplitude modulations were not lateralized but equally present at electrodes ipsilateral and contralateral to the target. A comparison of N1 mean amplitudes (averaged across electrodes PO7 and PO8) in the 150–200 msec time interval after search display onset between blocks with high and low vWM load confirmed that the reduction in the size of N1 components in high vWM load blocks was significant (M = −4.22 vs. −8.40 μV; t(13) = 5.30, p < .001). We also assessed whether an analogous load-induced N1 attenuation would also be present for ERPs triggered by memory test displays after the retention period. Figure 3 (left) shows ERPs at lateral posterior electrodes (averaged across PO7 and PO8) in response to memory test displays in low versus high vWM load blocks. N1 components were indeed smaller when vWM load was high, and this difference was again reliable (M = −5.48 vs. −8.05 μV; t(13) = 3.43, p < .005). To explore whether these reductions in the size of visual N1 components elicited by search and memory test displays with high vWM load were associated with task performance, we ran correlation analyses for N1 amplitudes for individual participants in high and low vWM load blocks and their performance in the search and memory tasks (RTs for the search task, error rates for the vWM task). The size of individual N1 components in high vWM load blocks predicted participants' memory performance in these blocks, with larger N1 components associated with more errors in response to memory test displays. This correlation was present both for N1 amplitudes elicited by search displays (r = −.533, p = .05) and also for N1 amplitudes triggered by memory displays (r = −.540, p = .046). There were no reliable associations between N1 amplitudes to search or memory test displays and memory performance in low vWM blocks. N1 amplitudes were also unrelated to participants' RTs in the search task.
Discussion of Experiment 1
The results of Experiment 1 provide new electrophysiological evidence that search-unrelated vWM load affects the efficiency of rapid template-guided attentional selection processes. The N2pc to color-defined target objects emerged approximately 70 msec later in blocks with high vWM load during the preceding retention period relative to low vWM load blocks. This suggests that maintaining multiple shapes in vWM impaired the preparatory activation of color-specific search templates, resulting in a delayed allocation of attention to template-matching target objects in search displays. Target N2pc amplitudes were also smaller in high vWM load blocks, indicating that template-guided attentional biases for target objects were triggered less strongly in these blocks. A further unexpected finding was that increasing vWM load in Experiment 1 also reduced the amplitude of the earlier visual-evoked N1 component in response to both search and memory test displays. Notably, the size of N1 components in high vWM load blocks was correlated with participants' individual performance in the vWM task, with larger N1 amplitudes to both search and memory test displays associated with more incorrect memory-matching responses. As visual N1 components are sensitive to manipulations of selective attention (e.g., Eimer, 1994; Mangun & Hillyard, 1991), the link between N1 amplitudes to displays presented at the end of the retention period and memory performance could reflect an attentional trade-off between memory maintenance and the perceptual processing of new visual input (see General Discussion). Because these N1 amplitude effects were not predicted, their replicability needs to be confirmed first. One goal of Experiment 2 was to provide such a confirmation.
In contrast to the clear effects of high versus low vWM load on target N2pc components, there was no corresponding behavioral load effect on search performance, as target RTs were only numerically but not reliably slower in high as compared with low vWM load blocks. This is surprising, as a delay of template-guided attentional target selection processes should presumably result in an increase of RTs to target objects. The absence of significant behavioral load effects in Experiment 1 could be due to the fact that vWM load was blocked in this experiment, with five low-load blocks followed by five high-load blocks, or vice versa. Any effects of vWM load may therefore have been masked by general practice effects resulting in faster responses in the second half of the experiment, in particular for participants who completed low vWM load blocks first. This possibility was addressed in Experiment 2, where trials with high and low vWM load were randomly intermixed.
Importantly, the effects of high versus low vWM load on target N2pc components in Experiment 1 were observed in spite of the fact that target colors remained constant throughout the entire experiment, and the corresponding color templates could therefore in principle have been transferred to a different longer-term memory store (e.g., Woodman et al., 2007). This suggests that multiple feature templates are maintained in vWM, even when they remain unchanged (cf. Grubert et al., 2016). In contrast, an attentional template for a single target feature (e.g., a specific color) may be transferred to a long-term memory store when this feature stays constant (Woodman et al., 2007). If this is correct, increasing vWM load should not affect template-guided attentional target selection processes (as reflected by target N2pc components) when observers search for a single color-defined target object. This prediction was tested in Experiment 2. The vWM task was the same as in Experiment 1, except that memory sample displays with one or four shapes (low vs. high vWM load) now appeared with equal probability and in random order within each block. There were two search tasks. The two-color task was identical to Experiment 1. In the one-color task, a single target color remained task-relevant throughout. If multiple constant target templates are maintained in vWM whereas single constant templates are held in a different memory store, increased vWM load should impair template-guided target selection processes only in the two-color task but not in the one-color task.
Fifteen individuals took part in Experiment 2. One participant's data were excluded due to a large number of artifacts produced by eye movements (>80% of all EEG epochs). The remaining 14 participants (M age = 29 years, SD = 6 years; six men; one left-handed) all reported normal or corrected-to-normal vision.
Stimuli and Procedures
Stimuli and procedures were similar to Experiment 1, with the following exceptions. Trials with high and low vWM load were now randomly intermixed within blocks. There were two versions of the search task, each presented in 10 successive blocks. The two-color task was identical to Experiment 1, with targets in the search displays defined by one of two possible and equally likely colors. In the one-color task, these targets were defined by a single color that remained constant throughout. This target color was determined randomly for each participant, with the exception that it was never identical to one of the target colors in the two-color task. Gray never served as target color in either task. Each block contained 36 trials (16 trials starting with a low vWM load memory sample display and 16 trials starting with a high-load vWM sample). On 24 trials, a search display was presented after the retention period, and on 12 trials, a memory test display was shown instead. Seven participants completed 10 one-color task blocks before the 10 two-color task blocks, and this order was reversed for the other seven participants.
EEG Recording and Data Analysis
Procedures were identical to Experiment 1, except that ERPs were computed separately for the one-color and two-color tasks. The average general EEG data loss due to artifacts and response errors was 12% (SD = 16). The absolute N2pc onset criteria used in the jackknife-based analyses were again defined as 50% of the N2pc peak amplitude for high vWM load blocks, separately for the one-color and two-color tasks. Onset latencies were compared within factorial analysis, with F values corrected according to the formula described by Ulrich and Miller (2001). This resulted in absolute thresholds of −0.72 and −0.55 μV for these two tasks, respectively. Based on the observations of Experiment 1, additional planned analyses were now conducted to assess the effects of vWM load on mean amplitudes of the N1 component, measured in the 150–200 msec poststimulus time window. These analyses were conducted separately for N1 components to search displays and to memory test displays. In addition, correlational analyses explored the link between load-dependent N1 amplitude modulations and individual vWM performance.
Correct RTs and error rates for the visual search task were entered into 2 × 2 repeated-measures ANOVAs with the factors Search task (one-color task, two-color task) and vWM load (low, high). RT data showed a main effect of Search task, F(1, 13) = 33.62, p < .001, ηp2 = .72, with slower RTs to targets in the two-color as compared with the one-color task (M = 703 vs. 652 msec). There was no main effect of vWM load (F < 1), but a significant interaction between vWM load and Search task was observed, F(1, 13) = 6.21, p < .03, ηp2 = .32. In the one-color task, there was no effect of low versus high vWM load on RTs to search targets (M = 653 vs. 651 msec; t < 1). In contrast, target RTs in the two-color task were significantly delayed on trials with high vWM load relative to low vWM load trials (M = 710 vs. 696 msec; t(13) = 2.82, p < .02). Error rates in response to search displays varied between 2% and 4% in different task conditions. There were no significant main effects of Search task, F(1, 13) = 2.58, p > .10, or vWM load (F < 1), and no interaction between these factors (F < 1). An ANOVA of error rates in the memory-matching task obtained a main effect of vWM load, F(1, 13) = 43.34, p < .001, ηp2 = .77, with more errors on trials with high versus low vWM load (M = 20% vs. 3%). There was no effect of Search task and no interaction between vWM load and Search task for error rates (both Fs < 1).
Figure 4 shows ERPs triggered by search displays at electrodes PO7/PO8 contralateral and ipsilateral to targets and the corresponding N2pc difference waveforms measured on trials with low and high vWM load in the one-color task (top) and in the two-color task (bottom). Target N2pcs were present in both search tasks. In the one-color task, N2pcs emerged at the same time on trials with low and high vWM load, but N2pc amplitudes appeared to be attenuated with high load. In the two-color task, the target N2pc was both delayed and smaller on trials where vWM load was high.
N2pc onset latencies determined with the jackknife-based method were evaluated in an ANOVA with the factors Search task and vWM load. There was a marginal effect of Search task, Fc(1, 13) = 3.60, p = .08, as target N2pcs tended to emerge later in the two-color relative to the one-color task (M = 276 vs. 253 msec). More importantly, a significant effect of vWM load was present, Fc(1, 13) = 4.69, p < .05, with delayed N2pc onsets on trials with high vWM load (M = 272 vs. 253 msec). Critically, this effect of vWM load on N2pc onset latencies differed between the one-color and two-color tasks (interaction between search task and vWM load: Fc(1, 13) = 3.91, p = .04, one-tailed). In the two-color task, target N2pcs were significantly delayed on trials with high versus low vWM load (M = 288 vs. 264 msec; tc(13) = 2.44, p < .03). In contrast, no N2pc onset difference between these two types of trials was present in the one-color task (M = 255 vs. 251 msec; tc < 1).
N2pc mean amplitudes were entered into a 2 × 2 × 2 ANOVA with the factors Search task, vWM load, and Laterality. This showed a significant main effect of Laterality, F(1, 13) = 13.76, p < .005, ηp2 = .51, indicating the reliable presence of N2pc components. There was also a Search Task × Laterality interaction, F(1, 13) = 5.22, p < .05, ηp2 = .29. Although reliable N2pc components were elicited in both tasks (ts > 2.86, ps < .02), N2pc amplitudes were larger in the one-color relative to the two-color task (M diff = −1.37 vs. −0.79 μV). A trend for a vWM Load × Laterality interaction, F(1, 13) = 3.28, p = .09, ηp2 = .20, reflected a tendency for N2pc amplitudes to be smaller on trials with high versus low concurrent WM load (M diff = −0.93 vs. −1.23 μV), although N2pc components were reliably present on both types of trials (ts > 2.82, ps < .02).
Figure 4 shows that nonlateralized N1 components were attenuated on trials with high vWM load as compared with low vWM load trials, both in the one-color and two-color tasks, analogous to Experiment 1. As can be seen in Figure 3 (right), the same was also the case for N1 components triggered by memory test displays in Experiment 2 (here shown collapsed across the one-color and two-color tasks). These load-induced N1 amplitude modulations were assessed by analyzing N1 mean amplitudes measured in the 150–200 poststimulus time window for search and memory test displays, respectively, with the factors vWM load and search task. Main effects of vWM load (reflecting reduced N1 amplitudes when vWM load was high) were found both in response to search displays, F(1, 13) = 57.26, p < .001, ηp2 = .82 (M = −4.23 vs. −7.29 μV), and memory test displays, F(1, 13) = 14.57, p < .005, ηp2 = .53 (M = −5.05 vs. −7.31 μV). There were no main effects of search task or interactions between both factors on N1 amplitudes for either type of display (all Fs < 1.3).
As in Experiment 1, additional correlation analysis across individual participants investigated links between these load-related N1 amplitude modulations and behavioral performance. Again, these modulations predicted memory-matching performance when vWM load was high. Participants who showed a larger N1 to search displays on trials with high vWM load tended to show poorer vWM matching performance on these trials, although this correlation was only marginally significant (r = −.495, p = .072). N1 amplitudes to memory test displays on high vWM load trials were reliably linked to memory-matching errors on these trials, with larger N1 amplitudes associated with more errors (r = −.630, p = .016). There were no correlations between the size of N1 amplitudes to search or memory test displays on trials with low vWM load and vWM performance on these trials, and no links between individual N1 amplitudes and participants' RTs in the one-color and two-color search tasks.
Discussion of Experiment 2
The N2pc onset latency differences observed in the two-color task confirmed the findings of Experiment 1. Target N2pc components were delayed when vWM load was high relative to low load trials, demonstrating that increasing the number of shapes maintained during the retention interval interfered with the activation of preparatory attentional templates for target colors and suggesting that these representations were held in the same WM store. Importantly, there was now also a reliable behavioral load effect, with RTs to search targets delayed on trials with high as compared with low vWM load. This suggests that the absence of a significant behavioral load effect in Experiment 1 was the result of vWM load being blocked rather than randomized across trials, as in Experiment 2.
Whereas clear load-induced interference effects were found for target selection in the two-color task, no such effects were observed in the one-color task. Here, increasing vWM load had no clear effect on target RTs or on target N2pc onset latencies. Because search performance and vWM maintenance were assessed on different trials, the absence of a load effect in this task could, in principle, be the result of participants failing to maintain multiple sample display items on trials where a search display was presented immediately after the retention period. However, these trials were randomly and thus unpredictably interleaved with trials where a memory test display was shown, and mean vWM accuracy was approximately 80% on the latter type of trials in both experiments. This makes it highly unlikely that the behavioral and electrophysiological effects of vWM load observed in response to search displays were affected by a selective failure to retain memory sample items. A more plausible alternative explanation for the absence of behavioral vWM load effects in the one-color task is that the target color template was no longer held in vWM, but in a different long-term memory store.
Target N2pc components also differed between the two tasks. They were reliably larger and tended to emerge earlier in the one-color relative to the two-color task. This is in line with previous N2pc results demonstrating that attentional target selection mechanisms operate more efficiently when they are guided by a single target template than by multiple templates (Grubert & Eimer, 2013). The fact that RTs to search targets were about 50 msec faster in the one-color as compared with the two-color task also supports this interpretation. It is notable that, in both experiments, target N2pc components emerged considerably later than in earlier studies that employed similar two-object search displays and analogous attentional selection tasks in the absence of concurrent search-unrelated vWM load. In an experiment where observers had to find a single or one of two possible color-defined targets in search displays where these targets were accompanied by a single distractor on the opposite side (Grubert & Eimer, 2013), target N2pcs emerged at about 180 msec (one-color task) or 210 msec (two-color task) after search display onset. In this study, the corresponding N2pc onset latencies for the one-color and two-color tasks were generally delayed by about 50 msec when vWM load was low and even more with high vWM load. These delays could reflect general dual-task costs for the control of visual search that arise whenever one or more search-unrelated items have to be concurrently maintained in vWM. If this was correct, target N2pcs should emerge much earlier under conditions where the same sample and search displays are presented, but participants are instructed to ignore the sample displays and only perform the search task on each trial. This hypothesis was tested in Experiment 3.
Similar to the first experiment, vWM load had a strong impact on visual N1 components triggered by search and memory test displays in Experiment 2, which were attenuated on trials with high vWM load. This N1 modulation was identical in the one-color and two-color tasks, demonstrating that it was determined entirely by the load of the vWM task, irrespective of whether one or two target color templates had to be maintained. The link between these load-related N1 amplitude modulations and individual memory performance observed in Experiment 1 was also confirmed: Participants who showed larger N1 components to search or memory test displays on trials with high vWM load performed worse when their shape memory was tested. However, the factors responsible for the load-induced modulation of N1 amplitudes observed in both experiments remain unclear. One possibility is that this effect is directly linked to the number of objects that are actively maintained in vWM. Alternatively, it may reflect an attenuated sensory response to search displays on trials where they were preceded by sample displays containing four objects.1 Multiple-item sample displays may trigger a stronger adaptation of visual neurons relative to single-item displays, and this could produce a reduction of N1 amplitude to subsequent search displays. If this was correct, the same N1 adaptation effects should also be observed under conditions where memory sample displays are task-irrelevant and do not have to be encoded into vWM.
Experiment 3 was conducted to test this prediction, as well as to investigate whether target N2pc components emerge earlier in the absence than in the presence of a concurrent vWM task. There were two task conditions. The combined search/vWM task was identical to the one-color task of Experiment 2. In the search-only task, the same one-item or four-item sample displays were followed by a search display on all trials. Here, participants were instructed to ignore all sample displays and focus exclusively on the detection of target objects in the search displays.
Thirteen individuals took part in Experiment 3. One participant was removed due to EEG artifacts affecting more than 45% of all trials, leaving a final sample of 12 (M age = 32 years, SD = 7 years; five men; one left-handed).
Stimuli and Procedures
These were similar to previous experiments, with the following exceptions. Participants completed two tasks. The combined task was identical to the one-color task of Experiment 2. In the new search-only task, participants were instructed to passively view the items in the memory sample displays without memorizing them, as they were task-irrelevant. Memory sample displays were followed by a search display on all trials, and participants had to report the orientation of the target bar defined by a particular color. This target color remained constant across both tasks for each participant and was randomized across participants. Trials with one or four items in the memory sample displays were randomly intermixed in both tasks. Each block contained 36 trials. Two blocks were run for the search-only task and three blocks for the combined task to equate the number of search displays shown in each task. Six participants first completed two successive search-only blocks before the three blocks for the combined task, and this order was reversed for the other six participants.
EEG Recording and Data Analysis
The average general EEG data loss due to artifacts and response errors was 11% (SD = 9). ERPs were computed separately for the search-only and combined tasks. All N2pc analyses were conducted on EEG data averaged across trials with one and four items in the sample displays. This was done to improve signal-to-noise ratio, and because Experiment 2 had found no target N2pc onset latency differences between trials with high versus low vWM load in the one-color task. The absolute N2pc onset criterion used in the jackknife-based analyses was defined as 50% of the N2pc peak amplitude measured in the combined task, resulting in an onset threshold of −0.69 μV. Because this onset analysis showed that the N2pc to search targets emerged much earlier in the search-only task relative to the combined task, N2pc mean amplitudes were computed within different time windows for these tasks (260–360 msec after search display onset in the combined task, as in Experiments 1 and 2, and 200–300 msec poststimulus in the search-only task). N1 mean amplitudes in response to search displays were again computed within a 150–200 msec poststimulus time window, separately for both tasks and for trials with one versus four items in the sample displays.
Correct RTs for the visual search task were entered into a 2 × 2 ANOVA with the factors Task (search-only, combined) and Number of samples (one, four). Responses to search targets were much faster in the search-only task than in the combined task (M = 548 vs. 646 msec), reflected by a main effect of Task, F(1, 11) = 110.12, p < .001, ηp2 = .91. There was no main effect of Number of samples and no interaction between both factors (both Fs < 1). Error rates in the search task varied between 2% and 3% in different task conditions, and there were no significant effects (Fs < 1.69, ps > .20). As expected, participants' memory-matching performance in the combined task was much less accurate on trials with four sample items (error rates for high vs. low vWM load: M = 20 vs. 1%; t(11) = 7.70, p < .001).
Figure 5 (top) shows ERPs elicited by search displays at electrodes PO7/PO8 contralateral and ipsilateral to targets and the corresponding N2pc difference waves, separately for the combined task and the search-only task. Although target N2pc onset latency in the combined task was similar to the one-color task of Experiment 2, N2pc components emerged much earlier in the search-only task. This was confirmed by the jackknife-based N2pc latency analysis, which revealed a reliable onset difference between both tasks (180 msec after search display onset in the search-only task vs. 271 msec in the combined task, tc(11) = 2.85, p < .02). The analysis of N2pc mean amplitudes (obtained during the 200–300 msec and 260–360 msec poststimulus time windows in the search-only and combined tasks, respectively) showed a significant main effect of laterality, F(1, 11) = 9.37, p < .02, ηp2 = .46, and an interaction between laterality and task, F(1, 11) = 7.31, p < .05, ηp2 = .40, as N2pc components were larger in the search-only than in the combined task (M diff = −1.40 μV vs. −0.89 μV). Follow-up paired t tests investigating whether contralateral ERPs were more negative than ipsilateral ERPs confirmed that N2pcs were reliably present in both tasks (ts > 2.11, ps < .03, one-tailed).
Figure 5 (bottom) shows N1 components triggered by search displays in both tasks, separately for trials with one versus four sample display items. An attenuation of N1 amplitudes to search displays following a four-item sample display was present in both tasks but appears considerably larger in the combined task. This was confirmed by the analysis of N1 mean amplitudes with the factors Task and Number of samples. There was no main effect of Task (F < 1) but a highly significant effect of Number of samples, F(1, 11) = 56.17, p < .001, ηp2 = .84, as N1 amplitudes were smaller on trials with four sample items (M = −4.00 vs. −6.06 μV). Importantly, there was an interaction between both factors, F(1, 11) = 5.93, p < .05, ηp2 = .35, as this reduction of N1 amplitudes was twice as large in the combined task relative to the search-only task (M diff = 2.77 vs. 1.35 μV). Follow-up analyses confirmed that this N1 reduction by four versus one preceding sample item(s) was reliably present in both tasks (ts > 4.90, ps < .005).
Discussion of Experiment 3
In Experiment 3, N2pc components to search targets emerged much earlier in the search-only task than in the combined task where participants maintained one or four shapes in vWM during the interval before the onset of a search display. The N2pc onset observed in the search-only task (180 msec poststimulus) was identical to the N2pc onset latency found in an earlier single-task experiment where similar two-item search displays were used (Grubert & Eimer, 2013). In the combined task, the target N2pc was delayed by about 90 msec and emerged at a similar latency as in the one-color task of Experiment 2. These N2pc onset latency differences between the two tasks demonstrate that a concurrent vWM task has substantial costs for the speed of allocating attention to search target objects, even when participants search for a single target-defining feature.
The other main finding of Experiment 3 concerns the nature of the N1 amplitude attenuations found in Experiments 1 and 2 on trials with high vWM load. The fact that this effect remained present in the search-only task indicates that it partly reflects sensory adaptation by multiple-item sample displays that is elicited even when these items do not have to be retained in vWM. However, the N1 amplitude modulation was significantly larger in the combined task, which suggests that active vWM maintenance processes also contribute to this effect. This will be further considered below.
The question whether and under which conditions attentional templates for visual search are maintained in vWM is still under debate. Some have argued that such templates are always held in vWM (e.g., Olivers et al., 2011), whereas others have claimed that, in tasks where search targets remain constant, target templates are rapidly transferred to a different long-term memory store (e.g., Carlisle et al., 2011). Here, we employed electrophysiological markers of template-guided attentional target selection processes to investigate how the load of a concurrent search-unrelated vWM task affects the efficiency of these processes. If search templates are held in vWM, increasing vWM load should interfere with the activation of these templates during the preparation for search and thus with their role in guiding attention toward target objects in search displays. Our results demonstrate that vWM load modulates the speed with which attention is allocated to target objects during relatively early visual processing stages. In Experiments 1 and 2, N2pc components to search target objects were delayed when observers had to maintain four as compared with just one shape during the retention period just before the presentation of a search display. This was the case both when vWM load was blocked (Experiment 1) or varied unpredictably across trials (Experiment 2). This target N2pc delay with high vWM load was observed in tasks where observers searched for one of two possible color-defined target objects, indicating that when two color-specific attentional templates have to be activated, these templates are held in vWM and thus interfere with other items that are simultaneously maintained in the same store.
It is notable that vWM load delayed target N2pc components in Experiments 1 and 2 in spite of the fact that target-defining colors remained constant for each participant throughout the entire experimental session. However, Experiment 2 demonstrated that this was the case only for the two-color search task. In the one-color task where observers always searched for the same color-defined target and target selection could therefore be guided by a single color template, there were no clear effects of high versus low vWM load on behavioral or electrophysiological markers of target selection. The presence of such effects in the two-color task and their absence in the one-color task suggest that attentional templates for a single target-defining feature can be transferred from vWM to a different long-term store when this template remains constant. However, multiple templates for different possible target features are maintained in vWM, even when they remain unchanged (see also Grubert et al., 2016, for corresponding evidence from the CDA component). Thus, the suggestion that search templates for constant target features are generally maintained outside vWM (Woodman et al., 2007) would seem to require qualification, as this appears to be the case only for single but not multiple target features, at least when these features come from the same dimension. Situations where targets are defined by one of two different possible target colors may involve additional top–down control processes, such as the activation of one color template and suppression of the other once a particular target object is encountered or switching between templates when target color changes across successive trials. None of these control processes is required during single-color search. It is important to note that the two-color search task employed in this study where a target could have one of two possible colors is different from a standard conjunction search task where targets are defined by a combination of two features (e.g., red circles). Future studies will need to investigate whether search templates for a single target object that is defined by a feature conjunction are moved from vWM to a different long-term store when they remain constant, in spite of the fact that these templates represent two different target features simultaneously.
The delay of target N2pc components with high as compared with low vWM load observed in the two-color tasks of Experiments 1 and 2 indicates that the storage of search-unrelated objects in vWM affects the time course of attentional allocation processes in visual search, with these processes initiated later when four objects as compared with a single object are concurrently maintained. Importantly, even when vWM load was low, target N2pc components also emerged substantially later in these two experiments than in previous N2pc studies of visual search without an additional vWM task. This suggests that the presence of such a task (regardless of its load) is sufficient to delay the start of target selection processes. Experiment 3 confirmed this prediction. Here, target N2pcs emerged at a typical latency of 180 msec poststimulus in the search-only task and were delayed by about 90 msec in the combined vWM/search task. This is consistent with previous behavioral results by Woodman et al. (2001), who found a general delay of RTs to search targets in dual search/memory tasks relative to search-only tasks. It is important to note that the delay of target N2pc components by a concurrent vWM task in Experiment 3 was found in the context of a one-color search task. In Experiment 2, increasing vWM load in this one-color task had no effect on N2pc onset latencies. Similarly, Woodman et al. (2001) also found no evidence for an additional delay of target selection processes when vWM load was increased in a task where participants searched for a single shape-defined target object. These observations and the pattern of N2pc latency differences found in this study suggest that a concurrent vWM task can affect the attentional selection of search targets in two different ways. On the one hand, there are dual-task costs relative to a search-only baseline that affect both single-feature and multiple-feature search tasks, independent of vWM load. These costs could be a generic effect of task expectancy. Participants may prepare less fully for the search task under conditions where search displays are only presented on two thirds of all trials and a memory test display on the other third. Incomplete task preparation could result in a delayed onset of template-guided target selection processes, irrespective of whether these templates are stored in vWM or in a different long-term memory store. On the other hand, there is the additional load-dependent delay of target selection processes that only emerged during multiple-feature search. This delay is likely to reflect the costs for the efficiency of attentional guidance by search templates that are produced by a competition between these templates and other items that are simultaneously maintained in vWM.
In Experiments 1 and 2, increasing the load of the vWM task from one to four items not only delayed N2pc components to search targets but also affected the amplitudes of the earlier visual evoked posterior N1 component elicited by the search and memory test displays, which were attenuated when vWM load was high. This load-induced N1 modulation was bilateral and thus independent of the position of target objects, which indicates that it represents a generic target-nonselective effect associated with the number of objects included in a memory sample display. Experiment 3 tested whether this effect simply reflects a stronger sensory adaptation of visual responses to search and test displays that were preceded by sample displays containing multiple objects or the number of items currently maintained in vWM. N1 components to search displays were smaller on trials where sample displays contained four objects even when these objects were task-irrelevant in the search-only task, indicating that sensory adaptation was involved. However, these N1 modulations were reliably larger in the combined task where the shapes in the sample displays had to be encoded into vWM. This suggests that this effect is at least partially associated with active maintenance mechanisms (see Rose, Schmid, Winzen, Sommer, & Büchel, 2005, for a similar modulation of visual N1 components that was associated with the number of objects held in WM in an n-back task). One possibility is that it reflects a competition for attention between vWM maintenance and the online perceptual processing of incoming visual events. Because attention can be allocated selectively either to external objects or to representations that are maintained internally (e.g., Chun, Golomb, & Turk-Browne, 2011), these two types of external versus internal attention compete with each other in dual tasks that involve both vWM maintenance and the selection of target objects in visual search displays. In such situations, directing attention to items held in vWM will impair the simultaneous allocation of external attention to new visual stimuli. This impairment will be greater when multiple objects are maintained in vWM. Corresponding behavioral evidence for such a competition between internal and external attention was found in a study by Konstantinou and Lavie (2013), where increasing vWM load impaired detection sensitivity for peripheral visual stimuli. The reduction of N1 amplitudes to search and memory test displays under high vWM load that was observed in this study is likely to reflect the electrophysiological effects of the same underlying competitive mechanisms. The observation that individual vWM performance was correlated with N1 amplitudes in the context of high vWM load in both experiments is in line with this interpretation. If there is a trade-off between external and internal attention and if different individuals prioritize these two types of attention differently, larger N1 components to search and memory test displays will reflect a tendency to allocate attention preferentially to new external events. Participants who have a tendency to prioritize external attention should show less accurate vWM performance, reflecting the limited availability of internal attention during vWM maintenance.
In summary, the current experiments provide new electrophysiological evidence for impaired template-guided attentional target selection processes in the presence of high concurrent vWM load. These impairments are likely to reflect the interference between preparatory search templates for multiple colors and other search-unrelated information that is concurrently held in vWM and thus support the hypothesis that such search templates are stored in vWM. The fact that only generic dual-task costs but no load-dependent interference effects were found during single-color search suggests that search templates for a single constant target-defining feature can be transferred from vWM to a different long-term store.
This work was supported by grant ES/L016400/1 from the Economic and Social Research Council (ESRC), United Kingdom. N. B. is also supported by a New Investigator grant (ES/R003459/1) from the ESRC.
Reprint requests should be sent to Nick Berggren, Department of Psychological Sciences, Birkbeck College, University of London, Malet Street, London WC1E 7HX, United Kingdom, or via e-mail: firstname.lastname@example.org.
We thank Geoff Woodman for suggesting this possibility.