Abstract

If we search for an item, a representation of this item in our working memory guides attention to matching items in the visual scene. We can hold multiple items in working memory. Do all these items guide attention in parallel? We asked participants to detect a target object in a stream of objects while they maintained a second item in memory for a subsequent task. On some trials, we presented this memory item as a distractor in the stream. Subjects did not confuse these memory items with the search target, as the false alarm rate on trials where the memory item was presented in the stream was comparable to that on trials with only regular distractors. However, a comparable performance does not exclude that the memory items are processed differently from normal distractors. We therefore recorded event-related potentials (ERPs) evoked by search targets, memory items, and regular distractors. As expected, ERPs evoked by search targets differed from those evoked by distractors. Search targets elicited an occipital selection negativity and a frontal selection positivity indexing selective attention, whereas the P3b component, which reflects the matching of sensory events to memory representations, was enhanced for targets compared to distractors. Remarkably, the ERPs evoked by memory items were indistinguishable from the ERPs evoked by normal distractors. This implies that the search target has a special status in working memory that is not shared by the other items. These other, “accessory” items do not guide attention and are excluded from the matching process.

INTRODUCTION

Visual search relies on the interaction between memory and vision. To find an item, we maintain a representation of this item in our working memory, and match this representation to the visual scene (Wolfe, 1994). In recent years, much has been learned about the neurophysiology of working memory. Many neurons that code the visual features of an item when it is perceived also maintain their activity if the item has to be remembered in the absence of the visual input. Persistent neuronal activity as a correlate of working memory has been observed in the frontal cortex (Rainer, Asaad, & Miller, 1998; Courtney, Ungerleider, Keil, & Haxby, 1997; Miller, Erickson, & Desimone, 1996; Wilson, O'Scalaidhe, & Goldman-Rakic, 1993), but also in many areas of the visual cortex (Corbetta, Kincade, & Shulman, 2002; Supèr, Spekreijse, & Lamme, 2001; Chafee & Goldman-Rakic, 2000; Chelazzi, Duncan, Miller, & Desimone, 1998; Miller & Desimone, 1994; Chelazzi, Miller, Duncan, & Desimone, 1993; Gnadt & Andersen, 1988; Fuster & Jervey, 1981). During visual search, the neuronal representation of the target in working memory (the “search template”; Duncan & Humphreys, 1989) has to interact with the incoming visual information in order to identify matching items and to specify their location. Models of visual search usually assume that the neurons representing the search template in higher areas provide a top–down signal to visual areas that enhances the activity of neurons responding to target features in the display (Hamker, 2005; Usher & Niebur, 1996; Desimone & Duncan, 1995; Bundesen, 1990; Phaf, Van der Heijden, & Hudson, 1990; Duncan & Humphreys, 1989). Neurophysiological studies in monkeys have confirmed this idea (Tomita, Ohbayashi, Nakahara, Hasegawa, & Miyashita, 1999). Neurons in the visual and frontal cortex have a stronger response if their receptive field falls on an item that the monkey is looking for compared to when it falls on a distractor (Bichot, Rossi, & Desimone, 2005; Chelazzi et al., 1993, 1998; Schall & Hanes, 1993). At a psychological level of description, the enhancement of neuronal firing rates is associated with a shift of visual attention to the matching item in the display. In short, working memory representations focus visual attention on matching items in the display (Desimone & Duncan, 1995; Wolfe, 1994; Bundesen, 1990; Phaf et al., 1990; Duncan & Humphreys, 1989).

The total number of items that can be stored in visual working memory is approximately four (Cowan, 2001; Luck & Vogel, 1997). In many cases, some of these items are not used immediately but only become relevant after a delay (Ballard, Hayhoe, & Pelz, 1995). It is an important theoretical question whether all items in working memory have a similar status. If so, the items in working memory that are stored for later use (hereafter called accessory memory items) should also guide attention to matching items in the display, just like the search template. A number of recent studies (Woodman & Luck, 2007; Houtkamp & Roelfsema, 2006a; Soto, Heinke, Humphreys, & Blanco, 2005; Downing & Dodds, 2004) investigated this question and found that the search template has a much stronger influence on the deployment of attention than the accessory memory items.

These results imply that items in visual working memory can attain at least two different states (see also Cowan, 2001). The first is that of the search template that guides attention to the matching items in the display. Accessory items, on the other hand, have less influence on the deployment of attention. Nevertheless, the aforementioned studies do not fully agree on the residual effects of these accessory items. Houtkamp and Roelfsema (2006a) and Downing and Dodds (2004) asked subjects to search for an item, while subjects held another item in memory for a subsequent task. They found that the accessory memory item caused little or no interference with visual search, even if there was a matching item in the display. Their results support a model of working memory where the accessory items are in an isolated state so that they cannot influence the deployment of attention (Figure 1A).

Figure 1. 

Three models of the organization of items in working memory. All models attribute a special status to the “search-template”: the item in working memory that provides top–down signals to visual areas to focus attention onto matching items in the display. (A) The accessory memory items are isolated from visual processing, as only the search template can influence the deployment of visual attention during search. (B) The accessory memory items attract attention, although to a lesser degree than the search template. (C) The subject strategically uses the accessory memory items as a “template for rejection.” The accessory memory items send top–down signals to visual areas that repel attention.

Figure 1. 

Three models of the organization of items in working memory. All models attribute a special status to the “search-template”: the item in working memory that provides top–down signals to visual areas to focus attention onto matching items in the display. (A) The accessory memory items are isolated from visual processing, as only the search template can influence the deployment of visual attention during search. (B) The accessory memory items attract attention, although to a lesser degree than the search template. (C) The subject strategically uses the accessory memory items as a “template for rejection.” The accessory memory items send top–down signals to visual areas that repel attention.

Soto et al. (2005), however, did observe attraction of attention by items stored for a later task. Their subjects searched for a tilted line among vertical lines in a display where all lines were embedded in colored shapes. In some trials, one of these shapes matched an accessory memory item (in color or shape), and reaction times were shortest if the target line appeared in this item. Likewise, the results of Olivers, Meijer, and Theeuwes (2006) revealed a similar attraction of attention by accessory memory items in an additional singleton paradigm. These results suggest an “attraction” model in which the accessory items exert a top–down influence so that corresponding display items attract attention (Figure 1B). However, another study obtained yet different results. Woodman and Luck (2007) found that visual search was faster when the accessory memory item appeared in the search array as a distractor, an effect which also was observed in one of the experiments by Downing and Dodds (2004). Based on this result, Woodman and Luck (2007) suggested that items in working memory other than the search template may even repel attention. Participants may strategically avoid attending the distractors that match the memory item in the search array, as if its representation in working memory can act as a “template for rejection” (Woodman & Luck, 2007) (Figure 1C).

Thus, although the previous studies agree that the accessory memory items have a different representation than the search template, their exact status remains unclear. At first sight, it seems difficult to reconcile the apparently conflicting results. We note, however, that the previous studies focused on reaction times and accuracy: measures that reflect the final outcome of a number of processing steps (e.g., Sanders, 1990). During visual search, display items undergo sensory processing, may or may not attract attention and may or may not be matched against the search template. It is therefore possible that processing of a distractor that matches an accessory memory item differs from the processing of regular distractors at more than one processing step, whereas these differences are not reflected in the subject's performance. The accessory memory item may, for example, interfere because it attracts attention, but this effect may be offset because it is also rejected faster by the matching process if it acts as a “template for rejection.”

In the present study, we recorded event-related potentials (ERPs) to gain insight in these component processes. Participants searched for a target object in a stream of objects, while they memorized a second object for a subsequent task (Figure 2). We compared ERPs evoked by the search targets, memory items, and regular distractors. Specifically, we measured the occipital selection negativity (OSN; Harter, Aine, & Schroeder, 1982; Previc & Harter, 1982) and the frontal selection positivity (FSP; Anllo-Vento, Luck, & Hillyard, 1998; Kenemans, Kok, & Smulders, 1993) as indices of selective attention, and the later P3b (or P300; hereafter referred to as P3) to probe the matching of stimuli to memory (for a review, see Kok, 2001).

Figure 2. 

Sequence of events during a trial. (A) In the encoding phase, we presented two objects that were followed by a number indicating which of the two objects was the target (ST = search target) in the subsequent search stream. The other object (MI = memory item) had to be remembered for a subsequent memory task. (B) The search stream was composed of three distractors and a “stimulus of interest” (i.e., the search target, memory item, or a distractor). Then a response cue appeared which instructed the participants to indicate whether the target had been presented in the stream. (C) We then presented either the memory item, the search target, or a distractor as a memory probe. The response cue instructed the participants to indicate whether this probe matched the memory item. Note that in this particular example, subjects should respond “no” in both tasks.

Figure 2. 

Sequence of events during a trial. (A) In the encoding phase, we presented two objects that were followed by a number indicating which of the two objects was the target (ST = search target) in the subsequent search stream. The other object (MI = memory item) had to be remembered for a subsequent memory task. (B) The search stream was composed of three distractors and a “stimulus of interest” (i.e., the search target, memory item, or a distractor). Then a response cue appeared which instructed the participants to indicate whether the target had been presented in the stream. (C) We then presented either the memory item, the search target, or a distractor as a memory probe. The response cue instructed the participants to indicate whether this probe matched the memory item. Note that in this particular example, subjects should respond “no” in both tasks.

METHODS

Participants

Nineteen healthy volunteers (7 men; mean age = 21.8 years) with normal or corrected-to-normal visual acuity were paid for their participation in this study. They were all right-handed as assessed by the Edinburgh Handedness Inventory (Oldfield, 1971) and gave their informed consent to participate. All procedures were approved by the ethics committee of the Faculty of Psychology of Maastricht University.

Stimuli and Design

The participants were tested in a soundproof and electrically shielded room while seated in front of a 17-in. CRT monitor (60 Hz refresh rate). Stimuli were presented and responses were recorded using the Presentation software package (Neurobehavioral Systems, San Francisco, CA). Trials began with the presentation of a fixation cross (1200 ± 200 msec) in the middle of a gray screen. Subsequently, two randomly chosen shapes were presented (1000 msec each with 150 msec in between, see Figure 2). We used six complex unnatural objects (Blurred Outlined Random Tetris Shapes; Bledowski et al., 2006; Linden et al., 2003) with a size of approximately 6.0° × 6.0°, presented at a random location within 2.1° of the fixation cross. These complex objects cannot be easily verbalized and are therefore likely to be retained in visual working memory. After 150 msec, a cue (a “1” or a “2” with equal probability) was presented for 1000 msec to indicate which of the two objects served as the search target. The other object had to be held in memory for a subsequent object-matching task. We presented both objects before the cue to avoid potential differences in encoding strategies for targets of the search and memory task. After a delay of 1000 ± 200 msec, we presented the search stream that consisted of a sequence of three randomly chosen distractors and the “stimulus of interest” (i.e., the search target, the memory item or another, regular distractor) that defined the stream type. Thus, every search stream contained four unique stimuli, including one “stimulus of interest” presented at a random position in the stream (we ensured that every position was selected equally often across trials). The “stimulus-of-interest” distractor did not differ from the other distractors, but we only used this distractor in our ERP analyses to equate the number of trials in the three conditions. We presented the stimuli for 800 msec with an interstimulus interval of 200 msec. After the stream, we presented a response cue and the participant indicated whether the target was present (left mouse button) or absent (right mouse button). Streams contained equally often the search target, the memory item or only distractors (31% each). A fourth type of trials were catch trials (7%), in which the search target was presented twice to encourage the participants to continue their search when they detected a target. The participants were instructed to press the scroll button of the mouse on these catch trials.

In the subsequent memory task, we presented the memory item (match) on 33.3% of the trials, the search target of the previous stream (mismatch) on 33.3% of the trials, or a distractor (mismatch) on the remaining 33.3% of trials as memory probe, for a duration of 800 msec. A response cue appeared after an interval of 200 msec, and the participants had to indicate whether the probe matched the memory item (or not) by clicking the left (right) mouse button. All combinations of search and memory task conditions occurred equally often. Participants received auditory feedback after each button response. Subjects heard a high tone (100 msec, 1000 Hz) after a correct response, a low tone (500 Hz) after an error, and a ringing sound when they failed to respond within 5 sec. In addition, we gave information about the subject's average reaction time and error percentage after each run.

Procedure

We instructed the participants to respond as fast and accurately as possible and not to speak, blink, or move their eyes away from the fixation cross during stimulus presentation. Prior to the EEG measurement, participants were trained on a task that was similar to the main task, except that the four stimuli of the search array were presented simultaneously (one object on a jittered position in each quadrant). Participants performed 48 trials of this task and they subsequently practiced the main task (58 trials). After this training session, participants performed four blocks (14 min each) of 58 trials, resulting in a total of 232 trials. The entire experiment, including the practice session and placement of the electrode cap, lasted about 2.5 hours.

Electrophysiological Recording and Data Analysis

The electroencephalogram (EEG) was recorded (sampling rate 250 Hz) from 29 tin scalp electrodes (extended International 10–20 System) with reference electrodes placed at the mastoids. Signals were collected using the left mastoid as reference and re-referenced off-line to the average activity at the mastoids. Horizontal and vertical electrooculograms were recorded with bipolar electrodes placed at the external canthi and above and below the left eye. Electrode impedance was kept below 5 kΩ for all electrodes.

We band-pass filtered the EEG data between 0.01 and 30 Hz (filter slope 12 dB/octave) and subtracted the baseline activity in a 100-msec prestimulus interval. We analyzed the responses in a time window from −200 to 950 msec relative to stimulus onset. Only trials on which subjects gave correct responses for both tasks were analyzed. Catch trials and trials with artifacts (samples exceeding ±75 μV) were excluded from subsequent analyses.

Electrode sites were collapsed into a frontal (Fp1, F3, F7, Fz, AFz, Fp2, F4, F8), centro-parietal (C3, CP3, P3, Cz, CPz, Pz, C4, CP4, P4), and occipito-temporal (T5, O1, Oz, T6, O2) region of interest (ROI; see Figure 4). The mean amplitude of the FSP (250–450 msec), OSN (250–350 msec), and the P3 (350–600 msec) were evaluated within, respectively, the frontal, occipito-temporal, and centro-parietal ROI. Mean amplitudes of the FSP, OSN, and P3 evoked by the search target and the memory item were compared to those evoked by a regular distractor using pairwise, two-tailed t tests. Two participants were excluded from the analysis because they had a low accuracy on the secondary memory task (<81%), in combination with a poor fixation during the search task.

RESULTS

The subjects' average accuracy was high in the search task (93.7% correct) as well as in the memory task (90.8%). To investigate whether subjects sometimes confused the accessory memory item with the search target, we compared the trials where the memory item appeared as a distractor in the stream to trials with only regular distractors not represented in memory (Figure 3). The probability of a false alarm on memory item trials (5.0%) was similar to that on distractor trials (4.1%) [t(16) = 1.14, p > .25]. Thus, the memory items behaved as normal distractors during the search task. Nevertheless, the performance on the memory task was very good, which indicates that the accessory memory item was well remembered during the preceding search task. These performance data, taken together, indicate that the subjects memorized both items, while they were able to keep the memory representation of the search target separate from the representation of the accessory memory item.

Figure 3. 

Mean accuracy for the different conditions in the search (left) and memory (right) task. Note that the search target is the target in the search task, and requires a target present response. The lower performance for search targets compared to memory items and distractors is presumably due to a response bias. Importantly, accuracy for memory items is similar to that for distractors. Because memory items and distractors both require a target absent response, performance on these trials can be compared. Finally, note that the memory item is the target in the memory task. In addition, the accuracy in the secondary memory task is specified as a function of the stimulus of interest in the preceding search task. Error bars indicate standard errors of the mean.

Figure 3. 

Mean accuracy for the different conditions in the search (left) and memory (right) task. Note that the search target is the target in the search task, and requires a target present response. The lower performance for search targets compared to memory items and distractors is presumably due to a response bias. Importantly, accuracy for memory items is similar to that for distractors. Because memory items and distractors both require a target absent response, performance on these trials can be compared. Finally, note that the memory item is the target in the memory task. In addition, the accuracy in the secondary memory task is specified as a function of the stimulus of interest in the preceding search task. Error bars indicate standard errors of the mean.

We next investigated whether the presence of the search target or memory items in the search stream influenced the performance in the subsequent memory task (Figure 3, right). The probability of a correct rejection of the search target in the memory task increased from 89.2% to 96.1% if it had appeared as a target in the preceding search stream [t(16) = 3.95, p < .002]. This suggests that subjects were able to refresh their memory representation of the search target if it appeared in the stream. In contrast, the presence of the accessory memory item in the search stream did not have a strong influence on performance in the memory task [t(16) = 1.81, p > .05].

ERPs Evoked by Targets, Accessory Memory Items, and Distractors

We based our ERP analysis on all trials where the subjects gave a correct response in both tasks (85.5% of the trials). We first investigated whether the amplitude of the ERP components interacted with the position of the critical item in the stream. We did not observe interactions between stimulus type and position in the stream, and we therefore pooled responses across all the positions in the stream for the subsequent analyses. Figure 4 shows the average ERP waveforms evoked by the search targets, the accessory memory items, and the regular distractors that were embedded in the search stream. The early ERP components (P1, N1, and P2) evoked at latencies up to 200 msec can be seen to be similar across all types of items, but at longer latencies the ERPs evoked by the targets differ from those evoked by the distractors.

Figure 4. 

Grand-average ERPs for the search target, memory item, and regular distractors presented in the search stream; shown for every scalp electrode site. Colored frames indicate assignment of the electrodes to one of three regions of interest (ROIs) for the subsequent analyses.

Figure 4. 

Grand-average ERPs for the search target, memory item, and regular distractors presented in the search stream; shown for every scalp electrode site. Colored frames indicate assignment of the electrodes to one of three regions of interest (ROIs) for the subsequent analyses.

The differences between target and distractor processing are also clearly evident in Figure 5, which displays grand-average ERPs at the frontal, centro-parietal, and occipito-temporal ROI. The target and distractor ERPs start to differ at a latency of approximately 250 msec above the frontal and occipito-temporal cortex, and this is followed by a later difference in response above the centro-parietal cortex. Importantly, the ERPs evoked by the memory items appear to be similar to those evoked by the distractors.

Figure 5. 

Average evoked potentials in the search task recorded from frontal, centro-parietal, and occipito-temporal electrodes. Note that the ERP evoked by the accessory memory items is similar to that evoked by regular distractors, whereas it differs from the ERP evoked by the search target.

Figure 5. 

Average evoked potentials in the search task recorded from frontal, centro-parietal, and occipito-temporal electrodes. Note that the ERP evoked by the accessory memory items is similar to that evoked by regular distractors, whereas it differs from the ERP evoked by the search target.

Our statistical analysis supports this description. Planned comparisons confirmed that the search target elicited a more negative response than the distractor at the posterior electrodes [OSN, 250–350 msec: mean difference = 1.71 μV; t(16) = 4.92, p < .0002], whereas it evoked a more positive response than distractors at the anterior electrodes [FSP, 250–450 msec: mean difference = 1.42 μV; t(16) = 2.15, p < .05]. In addition, the centro-parietal P3 was strongly enhanced for search targets compared to distractors [350–600 msec: mean difference = 3.30 μV; t(16) = 4.44, p < .0005].

In contrast, the response evoked by the memory items was similar to the response evoked by regular distractors. We did not observe significant differences in the ROIs in any of the time intervals (t tests, all ps > .19). This suggests that the processing of the memory items was similar to that of regular distractors, but different from the processing of search targets. Thus, items in the search stream matching the accessory memory item do not appear to attract attention and do not seem to be matched against the information in working memory. To ensure that any potential differences between memory items and distractors at individual frontal electrodes were not obscured by their joint analysis, we additionally submitted the mean amplitudes of the FSP (250–450 msec) to a repeated-measures analysis of variance with distractor type (memory item, distractor) and electrode (Fp1, F3, F7, Fz, AFz, Fp2, F4, F8) as factors. There was no interaction between the two factors [F(7, 105) = 0.32, p > .81; Greenhouse–Geisser corrected], indicating that the similarity between responses elicited by the memory item and a regular distractor did not differ between electrode sites.

In the secondary memory task, the roles of the search target and memory item were reversed. Now the memory item was a target while the former search target became a distractor. We expected that the memory items, which were treated as distractors during the search task, would now be processed differently. This expectation was confirmed by the data (Figure 6). In the secondary task, the memory item gave stronger responses than distractors in the FSP [frontal ROI; mean difference = 2.71 μV; t(16) = 5.23, p < .0001] and P3 [centro-parietal ROI; mean difference = 2.67 μV; t(16) = 5.41, p < .00006] latency range, but not in the OSN time window [occipito-temporal ROI; t(16) = 0.35, p > .7]. Conversely, the response evoked by former search targets resembled the response evoked by the distractors (t tests, all ps > .6).

Figure 6. 

Evoked potentials in the secondary memory task recorded from frontal, centro-parietal, and occipito-temporal electrodes. Now, the memory item ERP differs from the distractor ERP, whereas the search target ERP does not.

Figure 6. 

Evoked potentials in the secondary memory task recorded from frontal, centro-parietal, and occipito-temporal electrodes. Now, the memory item ERP differs from the distractor ERP, whereas the search target ERP does not.

DISCUSSION

In many tasks, we store items in memory for later use. Here we have investigated if and how these items influence the processing of visual stimuli. The data clearly indicate that the accessory memory items are stored in an inactive state with little influence on visual processing. At a behavioral level, we found that the false alarm rate caused by the memory items was similar to the rate caused by the regular distractors not in memory. Thus, our subjects were well able to memorize two similar items while using only one of them as a template during visual search. These results are in accordance with previous studies that also showed that accessory memory items cause little interference during visual search (Houtkamp & Roelfsema, 2006a; Downing & Dodds, 2004).

In addition, we addressed the possibility that there may exist differences between the processing of memory items and distractors that are not reflected by behavioral measures. Our study is the first to compare the neurophysiological responses evoked by memory items, search targets, and distractors. We obtained two neurophysiological results that have a straightforward interpretation. First, we observed a clear signature of the search template because the ERPs evoked by items in the stream that matched the search targets clearly differed from the ERPs evoked by distractors. Target and distractor ERPs differed during an early phase as well as during a later phase of processing. The frontal (FSP) and occipital (OSN) response differences are related to the selective processing of task-relevant stimuli (e.g., Anllo-Vento et al., 1998), whereas the later P3 component is thought to reflect the matching of stimuli to memory representations (variously interpreted as “context updating” [Donchin & Coles, 1988]; “event categorization” [Kok, 2001], and “template matching” [Chao, Nielsen-Bohlman, & Knight, 1995; Squires, Hillyard, & Lindsay, 1973]; for a review, see Kok, 2001).

Second, we found that the ERPs evoked by accessory memory items are indistinguishable from the ERPs evoked by distractors not in memory. Our design was sensitive to the difference in processing between targets and distractors, and the absence of a difference between memory items and distractors therefore implies that the accessory memory items do not share those properties with the search template in memory that cause targets in the search stream to be processed differently from distractors. In other words, the memory items do not guide attention and are not matched against the incoming visual information. Our results therefore support a model of working memory organization where the accessory memory items are in an isolated state and do not feed back to visual cortical areas to influence processing (as schematically indicated in Figure 1A). In contrast, the search template appears to occupy a special “slot” or partition in working memory. It is the only item that guides attention and that is matched against the incoming visual information (see also Houtkamp & Roelfsema, 2006a; Downing & Dodds, 2004).

Blockade by the Search Template

In the Introduction, we reviewed a number of studies that reached completely different conclusions regarding the influence of the accessory memory items on visual processing. Olivers et al. (2006) and Soto et al. (2005) found that display items matching an accessory memory item attracted attention, whereas Woodman and Luck (2007) observed that subjects used the accessory items in working memory to strategically avoid processing of irrelevant display items. How can we reconcile these findings with the studies that observed little influence of the accessory memory items, including the present one? A critical difference between studies that did and did not observe an influence of the memory items on visual processing is related to the type of search task. The studies that did not observe effects of the accessory memory items selected the search target from the same set of items as the memory items, and changed the identity of the search target from one trial to the next. These search targets therefore had to be stored in working memory at the beginning of every trial. In contrast, the subjects in the study of Woodman and Luck (2007), Olivers et al. (2006), and Soto et al. (2005) always searched for the same targets. These studies therefore used a so-called consistent mapping design (see Schneider & Shiffrin, 1977; Shiffrin & Schneider, 1977), where the search target comes from a set of items that is disjoint from the set of potential distractors. After sufficient training, the set of search targets becomes so familiar that it is unnecessary to show the target at the beginning of a trial, and it may therefore cease to occupy storage space in working memory (Woodman, Luck, & Schall, 2007). Consistent mapping changes a search task into a categorization task, and Schneider and Shiffrin (1977) showed that subjects become very efficient if they only have to decide whether one of the display items belongs to a fixed target category.

Figure 7 illustrates how the distinction between consistent and varied mapping conditions permits a reconciliation of present and previous results. If the search template occupies space in working memory, then it blocks the influence of the accessory memory items on visual processing so that they are in an isolated state (Figure 7A). However, if the search target remains the same across trials and ceases to occupy space in working memory, this blockade is released (Figure 7B), and now the accessory memory items may interfere with visual search (Olivers et al., 2006; Soto et al., 2005) or may even be used strategically as a template for rejection to make visual search more efficient (Woodman & Luck, 2007). Findings of Olivers (in press) and Oh and Kim (2003) confirm this interpretation. In these studies, participants searched for either an item that occupied space in working memory because it varied from trial to trial or for a vertically symmetric object or popout stimulus that did not occupy space in memory. The accessory memory item only interfered with the visual search when the subjects searched for an item that did not occupy memory space while interference was absent when it had to be memorized, which is in accordance with the model of Figure 7. Moreover, the model receives additional support from a recent study by Woodman et al. (2007). Their subjects performed a visual search for a constant target or for a target that changed from trial to trial. In the meantime, the subjects had to either remember four colors or none. This additional memory load interfered with visual search, but only if the search target was varied across trials, in accordance with the idea that constant targets do not occupy space in working memory.

Figure 7. 

How the search template may influence the effect of accessory memory items. (A) Storage of a search template blocks the influence of the accessory memory items on visual processing, and these items are therefore in an isolated state. (B) In the absence of a search template, the accessory memory items may gain a small influence on visual processing.

Figure 7. 

How the search template may influence the effect of accessory memory items. (A) Storage of a search template blocks the influence of the accessory memory items on visual processing, and these items are therefore in an isolated state. (B) In the absence of a search template, the accessory memory items may gain a small influence on visual processing.

Different States in Working Memory

The classical theory of Baddeley (2000) proposed that working memory is not a unitary system, but that it consists of several memory stores one of which is visual short-term memory. The present results indicate that there also exists a compartmentalization within visual short-term memory. The capacity of visual short-term memory is about four items (Cowan, 2001; Luck & Vogel, 1997), but all the items residing in working memory are not equivalent. There is a special “slot” in working memory for the search template: the item in memory that guides attention during visual search and that is matched against the items in the visual display. It is not yet clear whether more than one item in working memory can act as a search template. In the present study, it was a good strategy for the subjects to maintain the accessory memory items in an inactive state because if a matching item appeared in the stream, then it was a distractor. In a recent study, Houtkamp and Roelfsema (2006b) asked subjects to look for multiple target items in a stream of rapidly presented stimuli. The subjects of this study were unable to use more than a single template at a time, which suggests that the search template is a unique location in working memory that can hold, at most, a single item. This interpretation is consistent with the results of a seminal study of Sternberg (1966), who investigated how much time it takes for subjects to match a single character in a visual display to a varying number of characters in memory. Sternberg found that the reaction time of the subjects increased linearly with the number of memory items, as if the characters in memory were serially compared to the visually presented character. In the model of Figure 7, such a serial comparison entails a successive switching of items between the accessory memory buffer and the search template.

To gain insight in the neuronal representation of the search template, it is useful to briefly review the “biased competition” model of Desimone and Duncan (1995) (see also Miller & Cohen, 2001), who proposed that visually presented items engage in a competition in visual cortical areas. This model holds that the search template stored in the frontal cortex provides a top–down biasing signal to the lower visual areas to enhance the representation of visual items matching the search template, in order to increase the probability that these items win the competition and are selected by visual attention. This model has been supported by a number of neurophysiological studies in macaque monkeys. First, it has been shown that neurons in the frontal cortex of monkeys encode the search template (e.g., Miller et al., 1996). These cells are activated when the search target is presented, and they maintain their activity during memory episodes when the search target can no longer be seen. According to the theory, these neurons would act as a source for a top–down biasing signal to lower visual areas. Other studies recorded from neurons in areas of the visual cortex during visual search (Bichot et al., 2005; Chelazzi et al., 1993, 1998), and demonstrated that neurons activated by visual items that match the search template indeed enhance their activity. Here we observed that search targets evoked an OSN over the occipito-temporal cortex, that is, they generated a feedback signal, whereas the accessory memory items did not.

The results reviewed above support the idea that the search template is represented by the persistent activity of neurons in the frontal cortex. But how are the accessory memory items maintained? A recent EEG study by Vogel and Machizawa (2004) demonstrated that the storage of multiple items in working memory is associated with extra persistent neuronal activity. A higher memory load was associated with an increase in negativity over the contralateral occipital and parietal cortex. Correspondingly, fMRI studies have demonstrated that the hemodynamic response in frontal and parietal areas increases with memory load (e.g., Cohen et al., 1997), although this (depending on the brain area; Linden et al., 2003) levels off when memory capacity is exceeded (Todd & Marois, 2004). Only a few studies have investigated the activity of single neurons in monkeys during tasks where multiple objects have to be memorized. Warden and Miller (2007) showed that the persistent activity of neurons in the prefrontal cortex during a memory delay can code the identity of two objects. A surprising result of this study was that the representation of the first object in memory changed drastically when a second object was loaded. Taken together, these results suggest that the accessory memory items are also stored by the persistent firing of neurons in a distributed memory network. The representation of the accessory memory items therefore resembles the representation of the search template, and it will be important to investigate if there are differences between the topologies of the memory networks for the search template and the accessory items that can explain why only the former is matched to the incoming stimuli.

If there is only a single slot in working memory that can be matched to the visual input, then the item in this slot has to be replaced when the subject starts to look for another item. Our study required such a switch in the interval between the search stream and memory task, when the accessory memory item became target and the search target became distractor. At a neurophysiological level of description, the change of status of the accessory memory into a template for matching might require the activation of other neurons that can send feedback to lower visual areas. Previous studies suggested that the frontal cortex plays an important role in such a status change. Rao, Rainer, and Miller (1997), for example, instructed monkeys to first remember the identity of a search target and then presented a search display. Next, the animals had to memorize the location where the target item had been found. Many neurons in the frontal cortex coded the search template representing the target identity in the first part of the trial and, subsequently, coded the location of the matching visual item, in accordance with the changing task demands. This implies that some neurons in the frontal cortex code those attributes that are of current relevance.

The hypothesis that the frontal cortex is important for switching between accessory memory items and the search template is supported by a recent study by Soto, Humphreys, and Heinke (2006). These authors investigated the effect of accessory memory items on visual search in a group of patients with lesions in the frontal lobe. The patients searched for a tilted line among vertical distractors (i.e., the search template itself presumably needed little storage space) while they kept an item in working memory for a subsequent task. Interestingly, the frontal lobe patients suffered from a stronger interference of the memory items than control subjects. This suggests that the frontal cortex plays an important role in the storage of the accessory memory items in a format that prevents interference.

We conclude that the present study, together with other recent studies, provide a new understanding of the substructure of visual working memory. The search template is fed back to the visual cortex and evokes an OSN, whereas this is not the case for the accessory memory items. Future investigations can now start to address important questions about the neuronal mechanisms underlying the storage of the search template and the accessory items, and the different impact these representations in memory have on ongoing information processing in areas of the visual cortex.

Acknowledgments

We thank Nikolaus Kriegeskorte for designing the stimuli and Irene Nagel for assistance in some of the measurements. In addition, we gratefully acknowledge the useful comments on earlier versions of the manuscript by Bernadette Jansma and Joel Reithler. This research was supported by NWO grant 402-01-632 to J. C. P. and R. G.

Reprint requests should be sent to Judith Carolien Peters, Cognitive Neuroscience Department, Faculty of Psychology, Maastricht University, PO Box 616, 6200 MD, Maastricht, Netherlands, or via e-mail: j.peters@psychology.unimaas.nl.

REFERENCES

Anllo-Vento
,
L.
,
Luck
,
S. J.
, &
Hillyard
,
S. A.
(
1998
).
Spatiotemporal dynamics of attention to color: Evidence from human electrophysiology.
Human Brain Mapping
,
6
,
216
238
.
Baddeley
,
A.
(
2000
).
The episodic buffer: A new component of working memory?
Trends in Cognitive Sciences
,
4
,
417
423
.
Ballard
,
D. H.
,
Hayhoe
,
M. M.
, &
Pelz
,
J. B.
(
1995
).
Memory representations in natural tasks.
Journal of Cognitive Neuroscience
,
7
,
66
80
.
Bichot
,
N. P.
,
Rossi
,
A. F.
, &
Desimone
,
R.
(
2005
).
Parallel and serial neural mechanisms for visual search in macaque area V4.
Science
,
308
,
529
534
.
Bledowski
,
C.
,
Cohen Kadosh
,
K.
,
Wibral
,
M.
,
Rahm
,
B.
,
Bittner
,
R. A.
,
Hoechstetter
,
K.
et al
(
2006
).
Mental chronometry of working memory retrieval: A combined functional magnetic resonance imaging and event-related potentials approach.
Journal of Neuroscience
,
26
,
821
829
.
Bundesen
,
C.
(
1990
).
A theory of visual attention.
Psychological Review
,
97
,
523
547
.
Chafee
,
M. V.
, &
Goldman-Rakic
,
P. S.
(
2000
).
Inactivation of parietal and prefrontal cortex reveals interdependence of neural activity during memory-guided saccades.
Journal of Neurophysiology
,
83
,
1550
1566
.
Chao
,
L.
,
Nielsen-Bohlman
,
L. C.
, &
Knight
,
R. T.
(
1995
).
Auditory event-related potentials dissociate early and late memory processes.
Electroencephalography and Clinical Neurophysiology
,
96
,
157
168
.
Chelazzi
,
L.
,
Duncan
,
J.
,
Miller
,
E. K.
, &
Desimone
,
R.
(
1998
).
Responses of neurons in inferior temporal cortex during memory-guided visual search.
Journal of Neurophysiology
,
80
,
2918
2940
.
Chelazzi
,
L.
,
Miller
,
E. K.
,
Duncan
,
J.
, &
Desimone
,
R.
(
1993
).
A neural basis for visual search in inferior temporal cortex.
Nature
,
363
,
345
347
.
Cohen
,
J. D.
,
Perlstein
,
W. M.
,
Braver
,
T. S.
,
Nystrom
,
L. E.
,
Noll
,
D. C.
,
Jonides
,
J.
et al
(
1997
).
Temporal dynamics of brain activation during a working memory task.
Nature
,
386
,
604
608
.
Corbetta
,
M.
,
Kincade
,
J. M.
, &
Shulman
,
G. L.
(
2002
).
Neural systems for visual orienting and their relationships to spatial working memory.
Journal of Cognitive Neuroscience
,
14
,
508
523
.
Courtney
,
S. M.
,
Ungerleider
,
L. G.
,
Keil
,
K.
, &
Haxby
,
J. V.
(
1997
).
Transient and sustained activity in a distributed neural system for human working memory.
Nature
,
386
,
608
611
.
Cowan
,
N.
(
2001
).
The magical number 4 in short-term memory: A reconsideration of mental storage capacity.
Behavioral and Brain Sciences
,
24
,
87
114
.
Desimone
,
R.
, &
Duncan
,
J.
(
1995
).
Neural mechanisms of selective visual attention.
Annual Review of Neuroscience
,
18
,
193
222
.
Donchin
,
E.
, &
Coles
,
M. G.
(
1988
).
Is the P300 component a manifestation of context updating?
Behavioral and Brain Sciences
,
11
,
357
427
.
Downing
,
P. E.
, &
Dodds
,
C. M.
(
2004
).
Competition in visual working memory for control of search.
Visual Cognition
,
11
,
689
703
.
Duncan
,
J.
, &
Humphreys
,
G. W.
(
1989
).
Visual search and stimulus similarity.
Psychological Review
,
96
,
433
458
.
Fuster
,
J. M.
, &
Jervey
,
J. P.
(
1981
).
Inferotemporal neurons distinguish and retain behaviorally relevant features of visual stimuli.
Science
,
212
,
952
955
.
Gnadt
,
J. W.
, &
Andersen
,
R. A.
(
1988
).
Memory related motor planning activity in posterior parietal cortex of macaque.
Experimental Brain Research
,
70
,
216
220
.
Hamker
,
F. H.
(
2005
).
The reentry hypothesis: The putative interaction of the frontal eye field, ventrolateral prefrontal cortex, and areas V4, IT for attention and eye movement.
Cerebral Cortex
,
15
,
431
447
.
Harter
,
M. R.
,
Aine
,
C.
, &
Schroeder
,
C.
(
1982
).
Hemispheric differences in the neural processing of stimulus location and type: Effects of selective attention on visual evoked potentials.
Neuropsychologia
,
20
,
421
438
.
Houtkamp
,
R.
, &
Roelfsema
,
P. R.
(
2006a
).
The effect of items in working memory in the deployment of attention and the eyes during visual search.
Journal of Experimental Psychology: Human Perception and Performance
,
32
,
423
442
.
Houtkamp
,
R.
, &
Roelfsema
,
P. R.
(
2006b
).
Only one search template at a time.
Perception
,
35
, ECVP Abstract Supplement.
Kenemans
,
J. L.
,
Kok
,
A.
, &
Smulders
,
F. T.
(
1993
).
Event-related potentials to conjunctions of spatial frequency and orientation as a function of stimulus parameters and response requirements.
Electroencephalography and Clinical Neurophysiology
,
88
,
51
63
.
Kok
,
A.
(
2001
).
On the utility of P3b amplitude as a measure of processing capacity.
Psychophysiology
,
38
,
557
577
.
Linden
,
D. E.
,
Bittner
,
R. A.
,
Muckli
,
L.
,
Waltz
,
J. A.
,
Kriegeskorte
,
N.
,
Goebel
,
R.
et al
(
2003
).
Cortical capacity constraints for visual working memory: Dissociation of fMRI load effects in a fronto-parietal network.
Neuroimage
,
20
,
1518
1530
.
Luck
,
S. J.
, &
Vogel
,
E. K.
(
1997
).
The capacity of visual working memory for features and conjunctions.
Nature
,
390
,
279
281
.
Miller
,
E. K.
, &
Cohen
,
J. D.
(
2001
).
An integrative theory of prefrontal function.
Annual Review of Neuroscience
,
24
,
167
202
.
Miller
,
E. K.
, &
Desimone
,
R.
(
1994
).
Parallel neuronal mechanisms for short-term memory.
Science
,
263
,
520
522
.
Miller
,
E. K.
,
Erickson
,
C. A.
, &
Desimone
,
R.
(
1996
).
Neural mechanisms of visual working memory in prefrontal cortex of the macaque.
Journal of Neuroscience
,
16
,
5154
5167
.
Oh
,
S. H.
, &
Kim
,
M. S.
(
2003
).
The guidance effect of working memory load on visual search.
Journal of Vision
,
3
,
629a
.
Oldfield
,
R. C.
(
1971
).
The assessment and analysis of handedness: The Edinburgh inventory.
Neuropsychologia
,
9
,
97
113
.
Olivers
,
C. N. L.
(
in press
).
What drives memory-driven attentional capture? The effects of memory type, display type, and search type.
Journal of Experimental Psychology: Human Perception and Performance
.
Olivers
,
C. N. L.
,
Meijer
,
F.
, &
Theeuwes
,
J.
(
2006
).
Feature-based memory-driven attentional capture: Visual working memory content affects visual attention.
Journal of Experimental Psychology: Human Perception and Performance
,
32
,
1243
1265
.
Phaf
,
R. H.
,
Van der Heijden
,
A. H.
, &
Hudson
,
P. T.
(
1990
).
SLAM: A connectionist model for attention in visual selection tasks.
Cognitive Psychology
,
22
,
273
341
.
Previc
,
F. H.
, &
Harter
,
M. R.
(
1982
).
Electrophysiological and behavioral indicants of selective attention to multifeature gratings.
Perception & Psychophysics
,
32
,
465
472
.
Rainer
,
G.
,
Asaad
,
W. F.
, &
Miller
,
E. K.
(
1998
).
Selective representation of relevant information by neurons in the primate prefrontal cortex.
Nature
,
393
,
577
579
.
Rao
,
S. C.
,
Rainer
,
G.
, &
Miller
,
E. K.
(
1997
).
Integration of what and where in the primate prefrontal cortex.
Science
,
276
,
821
824
.
Sanders
,
A. F.
(
1990
).
Issues and trends in the debate on discrete vs. continuous processing of information.
Acta Psychologica
,
74
,
123
167
.
Schall
,
J. D.
, &
Hanes
,
D. P.
(
1993
).
Neural basis of saccade target selection in frontal eye field during visual search.
Nature
,
366
,
467
469
.
Schneider
,
W.
, &
Shiffrin
,
R. M.
(
1977
).
Controlled and automatic human information processing: I. Detection, search, and attention.
Psychological Review
,
84
,
1
66
.
Shiffrin
,
R. M.
, &
Schneider
,
W.
(
1977
).
Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory.
Psychological Review
,
84
,
127
190
.
Soto
,
D.
,
Heinke
,
D.
,
Humphreys
,
G. W.
, &
Blanco
,
M. J.
(
2005
).
Early, involuntary top–down guidance of attention from working memory.
Journal of Experimental Psychology: Human Perception and Performance
,
31
,
248
261
.
Soto
,
D.
,
Humphreys
,
G. W.
, &
Heinke
,
D.
(
2006
).
Dividing the mind: The necessary role of the frontal lobes in separating memory from search.
Neuropsychologia
,
44
,
1282
1289
.
Squires
,
K. C.
,
Hillyard
,
S. A.
, &
Lindsay
,
P. H.
(
1973
).
Vertex potentials evoked during auditory signal detection: Relation to decision criteria.
Perception & Psychophysics
,
14
,
265
272
.
Sternberg
,
S.
(
1966
).
High-speed scanning in human memory.
Science
,
153
,
652
654
.
Supèr
,
H.
,
Spekreijse
,
H.
, &
Lamme
,
V. A.
(
2001
).
A neural correlate of working memory in the monkey primary visual cortex.
Science
,
293
,
120
124
.
Todd
,
J. J.
, &
Marois
,
R.
(
2004
).
Capacity limit of visual short-term memory in human posterior parietal cortex.
Nature
,
428
,
751
754
.
Tomita
,
H.
,
Ohbayashi
,
M.
,
Nakahara
,
K.
,
Hasegawa
,
I.
, &
Miyashita
,
Y.
(
1999
).
Topdown signal from prefrontal cortex in executive control of memory retrieval.
Nature
,
401
,
699
703
.
Usher
,
M.
, &
Niebur
,
E.
(
1996
).
A neural model for parallel, expectation-driven attention for objects.
Journal of Cognitive Neuroscience
,
8
,
305
321
.
Vogel
,
E. K.
, &
Machizawa
,
M. G.
(
2004
).
Neural activity predicts individual differences in visual working memory capacity.
Nature
,
428
,
748
751
.
Warden
,
M. R.
, &
Miller
,
E. K.
(
2007
).
The representation of multiple objects in prefrontal neuronal delay activity.
Cerebral Cortex
,
17
,
i41
i50
.
Wilson
,
F. A.
,
O'Scalaidhe
,
S. P.
, &
Goldman-Rakic
,
P. S.
(
1993
).
Dissociation of object and spatial processing domains in primate prefrontal cortex.
Science
,
260
,
1955
1958
.
Wolfe
,
J. M.
(
1994
).
Guided Search 2.0: A revised model of visual search.
Psychonomic Bulletin & Review
,
1
,
202
238
.
Woodman
,
G. F.
, &
Luck
,
S. J.
(
2007
).
Do the contents of visual working memory automatically influence attentional selection during visual search?
Journal of Experimental Psychology: Human Perception and Performance
,
33
,
363
377
.
Woodman
,
G. F.
,
Luck
,
S. J.
, &
Schall
,
J. D.
(
2007
).
The role of working memory representations in the control of attention.
Cerebral Cortex
,
17
,
i118
i124
.