Recoding between Two Types of STM Representation Revealed by the Dynamics of Memory Search

Look at the desk in front of you. Now, close your eyes and try recalling the color of the pencil lying next to the paper pile on the right side of your desk. How do we recall such an object and its features? Do we reconstruct an exact picture in the “mind's eye” and recall its features by attending to the object like we do while categorizing a feature from a visual scene? Visual STM (VSTM) has been found to be closely related to visual attention (Akyürek, Leszczyński, & Schubö, 2010; Soto & Humphreys, 2006, 2009; Soto, Humphreys, & Heinke, 2006; Chelazzi, Duncan, Miller, & Desimone, 1998; Desimone, 1998; Desimone & Duncan, 1995). Although memory encoding and maintenance have long been shown to depend on attention mechanisms (Vogel, McCollough, & Machizawa, 2005; Vogel & Machizawa, 2004; Awh, Anllo-Vento, & Hillyard, 2000; Logan, 1988), recent studies have also linked orienting of attention to active search within VSTM (Dell'Acqua, Sessa, Toffanin, Luria, & Jolicœur, 2010; Eimer & Kiss, 2010; Kuo, Rao, Lepsien, & Nobre, 2009) and to subliminally stored objects (Astle, Nobre, & Scerif, 2010).

Modulations in attentional orienting to VSTM have been assessed by measuring the amplitude of attention markers—N2pc and N3rs. The N2pc component—a negative deflection observed over posterior contralateral electrodes 200 msec after stimulus presentation—has been found to reflect lateralized attention (Eimer, 1996; Luck & Hillyard, 1994). The N3rs—a negative deflection observed 300 msec after the presentation of a retro-(spective) search cue—component has been reported to be triggered by orienting attention toward locations within VSTM (Nobre, Griffin, & Rao, 2008). It has been reported in paradigms that use a so-called retro-search cue (with the cue indicating target location in a memory template) as a fronto-centrally distributed negativity starting 300 msec after the presentation of the retro-cue (Kuo et al., 2009; Nobre et al., 2008). The N3rs has been found to increase in amplitude with increasing VSTM load. Interestingly, the N3rs was modulated by neutral retro-cues only and not by spatial retro-cues (Nobre et al., 2008). This suggests that the N3rs might be susceptible to the amount of spatial information during memory search. Particularly, orienting attention to locations maintained in VSTM has been reflected in the amplitude of the N3rs, which is reduced for increased memory load (Nobre et al., 2008). Another component that has been associated with memory processing and which might be also susceptible to the amount of spatial information during memory search is the SPCN (sustained posterior contralateral negativity). The SPCN is also calculated as a lateralization index, and therefore, it might be expected to convey spatial information in VSTM, too. It is observed following encoding of spatial visual stimuli (Prime, Dell'Acqua, Arguin, Gosselin, & Jolicœur, 2010; Jolicœur, Brisson, & Robitaille, 2008). Particularly, it has been found to increase in amplitude with increasing memory load (Jolicœur et al., 2008), which makes it a possible marker of the amount of visuospatial memory search. Similar to the N2pc, it is calculated by subtracting activity measured at ipsilateral electrode sites from activity measured at contralateral electrode sites for lateralized presented targets. The SPCN has been thought to mark maintenance of information stored in STM (Jolicœur et al., 2008).

Investigating mechanisms of recall from VSTM after various delays might require recurring to the concept of STM consolidation (Jolicœur & Dell'Acqua, 1998). Consolidation has been found to operate rather in serial manner (Jolicœur & Dell'Acqua, 1998). Thus, it takes time for information maintained in VSTM to be fully consolidated. Varying retention interval (time lag between encoding and recall) might, therefore, correlate with the amount of successfully performed consolidation. The current study aimed at investigating ERP correlates of possible representational changes over short-term consolidation by manipulating the retention interval. Furthermore, STM consolidation has been found to involve central processes, which are also required for response selection, thereby causing delays (Jolicœur & Dell'Acqua, 1998). To investigate such behavioral effects, we also analyzed RT (cf., Jolicœur & Dell'Acqua, 1998).

The present study sought to address how attentional orienting in memory develops over time. In two experiments, we investigated the properties of recalled information. We used various retention intervals in a memory search task to cover the dynamics of the recall process. We devised a memory search paradigm to obtain an electrophysiological measure of attentional orienting strength within VSTM. Memory search was triggered by a centrally presented probe (see Figure 1). The primary measure was the amplitude of the N2pc elicited by the probe. We relate the magnitude of the measured amplitude to the degree of attentional orienting within VSTM. As a lateralized component, the N2pc is usually measured in relation to a laterally presented object. The N2pc in the current study was measured in response to a centrally presented probe (cf. Figure 1). As the probe in the present study was not lateralized, lateralization effects observed in this paradigm can be attributed to attentional orienting triggered by the probe but performed within VSTM. We additionally varied the duration of the retention interval between the presentation of the memory array and the probe ranging from 300 to 1700 msec. Varying the retention interval allowed us to measure the strength of attentional orienting within VSTM across time. On the basis of the observation that the amplitude of the N2pc in visual search is increased for conditions requiring more spatial attention (e.g., Eimer, 1996), we hypothesized that the strength of attentional orienting within the memory array would follow a similar pattern, that is, show a larger amplitude when more attention is needed for recall. We expected performance at short retention intervals to rely largely on the visuospatial representation of the stored information and to elicit larger N2pc (and N3rs) component amplitude. The weaker the attentional orienting to memorized array, the smaller component amplitude was expected to be. Decreasing amplitude with increasing retention interval would indicate gradual recoding from a visuospatial form into a new format.

In Experiment 2, we used a verbal working memory task to reveal the nature of this new representational format as it emerged over time (cf., Luck & Vogel, 1997). The verbal memory task should make recoding into a verbal format more difficult by engaging the articulatory loop. If VSTM representations are usually transformed into a verbal code over time, this would be prevented here, and the attention markers should be equally visible across retention intervals. In that case, recall would have depended solely on the visuospatial format.

Four color patches placed on a plane of invisible square (each stimulus placed 1° of visual angle away from center; each patch was a square extending 1° of visual angle in horizontal and vertical plane) served as stimuli in Experiments 1 and 2. Colors used in the sample were green (RGB 0, 255, 0), blue (RGB 0, 0, 255), red (255, 0, 0), and yellow (255, 255, 0). Participants responded with the index and middle fingers of the left and right hand using four keys. The keys were “w,” “s,” “k,” and “o” on a standard, international keyboard. Each of the keys was associated with a single color. The retrieval cues were assigned to an invisible “Z” plane starting with Position 1 in the top left corner of the “Z”; Position 2 in the top right corner, Pposition 3 in the bottom left corner, and Position 4 in bottom right corner, for half of the participants. For the other half the positions were placed on an inverted “Z” plane with Position 1 in the top right corner; Position 2 in the top left corner; Position 3 in the bottom right corner, and Position 4 in the bottom left corner.

Participants (n = 12, mean age = 25 years, SD = 2.5 years) viewed the memory array for 1500 msec and were asked to memorize the colors and their locations. After various retention intervals (300, 700, 1200, and 1700 msec), a centrally presented probe was flashed for 150 msec, symbolically indicating a location: A centrally presented digit number indicated a particular location. Locations were associated with numbers from a set of 1–4 during the practice block. Participants were asked to press as fast as possible (keeping accuracy above 90%) one of four buttons indicating the color of the square presented at the probed location. Pressing a response button triggered a feedback screen followed after 200 msec by the next trial. The experiment consisted of 14 blocks (64 trials each). In total, a participant performed 896 experimental trials. Before the experiment, a practice block of 64 trials had to be completed.

The EEG signal was recorded from 23 electrodes (sampling frequency of 500 Hz, 0.1–125 Hz bandwidth, referenced to Cz and re-referenced off-line to the average of all electrodes), subsequently the signal was low-pass filtered at 40 Hz and probe-locked. Epochs containing eye movements or eye blinks were excluded from analysis (for the procedure, see Wykowska & Schubö, 2010).

Separate repeated measures ANOVA were computed on the mean voltages time-locked to the centrally presented probe for the N2pc, SPCN, N3rs, and P3. The N2pc was measured from electrodes ipsilateral and contralateral to the probed item location in the memory template within a 220–280 msec time window at parieto-occipital electrode pairs (PO7–PO8 and PO3–PO4). The ANOVA included the factors electrode laterality (ipsilateral, contralateral to the position where the target had been presented in memory array), load (2 items, 4 items), retention interval (clustered short: 300 and 700 averaged together, clustered long: 1200 and 1700 averaged together), and electrode pair (PO7–PO8, PO3–PO4). A repeated measure ANOVA on the SPCN amplitude was measured within 300–400 msec poststimulus and also included the factor delay (300, 700, 1200, 1700 msec), laterality (contralateral, ipsilateral) and load (2 items, 4 items). The ANOVA for the N3rs measured 300-400 msec poststimulus over four (AFz, Fz, FCz, CPz) midline electrodes (cf. Nobre et al., 2008) and contained the factors electrode, load (2 items, 4 items), and retention interval (300, 700, 1200, 1700). A repeated measures ANOVA on P3 amplitude was measured within 280–380 msec poststimulus and included the factors delay (300, 700, 1200, 1700 msec) and electrode (Pz, POz; cf. Akyürek, Leszczyński, et al., 2010; Vogel, Luck, & Shapiro, 1998).

Figure 2A presents contralateral and ipsilateral activity measured at parieto-occipital electrodes. The main effect of Electrode Laterality [F(1, 11) = 6.58; p < .05] confirmed the presence of a pronounced N2pc as expected from previous research in visual search (Eimer, 1996) and memory search (Dell'Acqua et al., 2010; Eimer & Kiss, 2010; Kuo et al., 2009). Subtracting ipsilateral (3.21 μV) from contralateral (1.90 μV) activity resulted in a negativity (−1.31 μV), marking attentional orienting to the location where the target had been presented in the memory array. Notice that the N2pc was triggered by a centrally presented visual probe, which ensures that the negative deflection is only the result of orienting to a representation in memory and not orienting to any actual visual stimulation. This suggests that the centrally presented probe triggered a shift of spatial attention within VSTM. None of the further main effects nor the interaction of factors was reliable. Although memory orienting (as marked by the N2pc amplitude) was not affected by load, the N3rs component, which has been found to reflect retroactive target selection mechanisms (Kuo et al., 2009; Nobre et al., 2008), showed a greater negativity for Load 4 than for Load 2 [F(1, 11) = 20.00, p < .001]. Figure 2B plots mean amplitude of the N3rs component over representative midline electrodes. The main effect of load revealed that Load 4 was accompanied by a greater negativity than load 2 (−2.92 vs. −1.94 μV). This replicates a previously observed pattern (Kuo et al., 2009) with a load-dependent N3rs over midline electrodes (Figure 2B) accompanied by a load-independent N2pc (Figure 2A).

Figure 3A shows difference waves (contralateral minus ipsilateral) plotted separately for the four delays. The mean amplitudes and a potential interpretation are presented in Figure 3B and C, respectively. The negative deflection in the N2pc time window confirmed that observers were recalling by means of attentional orienting within VSTM. An interaction of electrode laterality and retention interval [F(1, 11) = 4.98, p < .05] showed that the amplitude of the N2pc was more pronounced at short delays than at long delays (−1.65 μV vs. −0.972 μV, respectively; see Figure 3A and B). The decrease of N2pc amplitude as a function of retention interval reflects a decrease in the strength of attentional orienting over time. It is worth to notice that there was no significant N2pc at longest intervals. The longer the interval between memory template presentation and the probe, the less spatial attention seemed required for recall. This leads us to conclude that the initial spatial format of the VSTM representation is gradually recoded into another form. We further computed the SPCN component, which revealed that laterality did not interact significantly with any other factor. However, post hoc comparison reflected there was a reliable SPCN component, but only at the shortest [t(11) = 2.41, p < .05] delay (300 msec: −1.48 μV; 700 msec: −0.80 μV; 1200 msec: −0.65 μV; 1700 msec: −0.83 μV). This could again reflect preserved spatial layout of memory representation at shortest retention interval and gradual recoding with increasing delay. Interestingly, the amplitude of the N3rs component (Figures 4A and 5A), thought to reflect retroactive spatial target selection (Kuo et al., 2009; Nobre et al., 2008), showed a similar pattern. It also decreased as a function of delay [F(3, 33) = 10.07, p < .001]. Post hoc contrasts revealed that there was a decrease in negativity of the N3rs component from delays of 300–1700 msec [F(1, 11) = 12.53, p < .005], 300–1200 msec [F(1, 11) = 14.03, p < .005], and 300–700 msec [F(1, 11) = 17.81, p < .001]. Mean amplitudes were −3.95 μV (300 msec), −2.31 μV (700 msec), −1.91 μV (1200 msec), and −1.56 μV (1700 msec). The decreasing N3rs negativity suggested that it becomes more and more difficult to select a spatial target from the memory array with increasing retention interval. These variations in difficulty could mirror a decrease in the degree of retroactive attentional orienting and retroactive spatial target selection (cf. Kuo et al., 2009; Nobre et al., 2008). Such an interpretation fits well with earlier findings of Nobre et al. (2008), where the N3rs was susceptible to the amount of spatial information conveyed by retro-cues. In the current experiment, the amount of spatial information seemed to be changing over retention intervals. Thus, the longer the retention interval between memory array and probe, the more difficult retroactive spatial target selection became. Taken together, results reflected by the N2pc and N3rs component indicate that the spatial memory representation is gradually being recoded into a new format, thus making attentional orienting more difficult as a recall strategy.

The effect of increasing difficulty for both attentional orienting in VSTM and retroactive spatial target selection were accompanied by a surprising pattern of RTs (see Figure 4C). RT decreased as a function of delay [F(3, 33) = 30.000, p < .005]. Post hoc contrasts revealed that participants were slower in responding with a delay of 300 msec than with any other delay. Significant decreases in RT were observed between the delays of 300 versus 700 msec [F(1, 11) = 21.228, p < .001] and between the delays of 1200 versus 1700 msec [F(1, 11) = 18.257, p < .001] but not between the delays of 700 and 1200 msec [p > .18]. Mean RTs were 855 msec (delay of 300 msec), 786 msec (delay of 700 msec), 771 msec (delay of 1200 msec), and 729 msec (delay of 1700 msec). Overall error rate was 8%. There was main effect of load [F(1, 11) = 17.45, p < .05] with Load 2 resulting in less errors than Load 4 [7% vs. 11%]. The main effect of retention interval was also significant [F(3, 33) = 4.91, p < .05]. Planned comparison revealed that the difference between delays of 1200 msec (11%) and 1700 msec (8%) was significant whereas the other differences were not (delay of 300 msec: 9% and delay of 700 msec: 7%). A significant interaction between both factors [F(3, 33) = 10.35, p < .001] indicated higher error rates for Load 4 at longer delays (14% for delay of 1200 msec and 9% for delay of 1700 msec) than at short delays (12% for delay of 300 msec and 7% for delay of 700 msec) and no variation in error rate for Load 2 (delay of 300, 700, 1200, and 1700 msec = 7%).

From the improvement in recall performance, one may conclude that the memory search task became easier with longer retention intervals. Given that the initial format of representation becomes gradually less susceptible to attentional orienting as recall strategy, we hypothesized that this initial memory representation was transformed into some other, less spatial form. This new representation facilitated memory recall as indicated by the decrease in RT. This claim was corroborated by the analysis of the P3 component (cf. Figure 4B). P3 amplitude increased as a function of delay [F(3, 33) = 9.447, p < .005]. Post hoc contrasts revealed a significant increase in positivity from 700 to 1200 msec [F(1, 11) = 13.520, p < .005] and from 1200 to 1700 msec [F(1, 11) = 8.766, p < .05], but not from 300 to 700 msec [p > .14]. The mean amplitude of the P3 was 2.715 μV (300 msec), 1.973 μV (700 msec), 3.220 μV (1200 msec), and 3.933 μV (1700 msec). Previous research has related the P3 component to updating or maintenance of information (Donchin & Coles, 1988). Attentional blink studies have associated the amplitude of the P3 to the amount of information that reaches memory (Vogel et al., 1998). Taken together, this could explain the current pattern. The maintenance or updating phase increased with increasing retention interval, in other words, the longer the retention interval, the more time participants had for maintenance. This would explain the observed increase in the P3 amplitude with longer delays. Furthermore, increasing P3 amplitude has also been related to the formation of a durable VSTM representation, a process also known as consolidation (Akyürek, Leszczyński, et al., 2010; Vogel et al., 1998). Given that the retention intervals varied from 300 to 1700 msec, the current P3 amplitude increase could be explained by the gradual formation of a durable VSTM representation, thereby marking the process of consolidation. The current pattern reveals how the initial spatial representation in VSTM may be replaced by another, nonspatial representation. This nonspatial representation seems to be developing during memory consolidation. Thus, the transition between the nature of the N2pc, SPCN, and N3rs effects to those of the P3/RT effects might be viewed as a correlate of gradual representational change from a spatial to a non-spatial memory representation. To examine the nature of the representation that is gradually built up during short-term consolidation, we conducted Experiment 2.

In Experiment 1, we found that the visuospatial format of VSTM was recoded into some other, less spatially-specific code. Experiment 2 was performed to reveal the nature of the recoded VSTM representation. Given that the stimuli in Experiment 1 were easy to verbalize, one could expect that the visuospatial format was recoded into a verbal representation. To test whether a verbal code is a possible format,¹ we used a verbal working memory task, which was adapted from Luck and Vogel (1997), to make it suitable for the EEG technique. This was done to increase the difficulty of recoding spatial VSTM contents into a verbal code. In case recoding of the visuospatial code into a verbal code was actually part of the participants' strategy to perform the task, then increasing recoding difficulty because of verbal working memory task should lead to a different pattern of visuospatial ERP markers as found in Experiment 1. If the N2pc–SPCN–N3rs pattern in Experiment 1 indicates gradual recoding into a verbal format then increasing the recoding difficulty should also affect the markers of memory search. Such modulations would indicate that loading verbal working memory changed the recall strategy used by participants. Additionally, if recoding into a verbal code is associated with efficient memory retrieval, the RT benefit for long retention intervals observed in Experiment 1 should also disappear in Experiment 2.

Twelve participants (mean age = 24.87 years, SD = 3.57 years; eight women) participated in Experiment 2. Unless stated differently, the method was the same as in Experiment 1. The VSTM task was embedded within a verbal working memory task. Participants were presented, via stereo loudspeakers, a verbal sequence of three letters (e.g., A, B, C sampled randomly without replacement from the set of four letters A, B, C, and D) at the beginning of each trial. After 1000 msec, a VSTM search task like in Experiment 1 was performed. Once participants responded to the VSTM task, they were asked to recall the verbal memory content by repeating the letters in correct order. The load condition was dropped in Experiment 2 (only load 4 was used) for efficiency reasons. Furthermore and to make verbal coding even more difficult, the memory array presentation time was set to 300 msec.

Separate repeated measures ANOVAs were computed on mean amplitudes time-locked to the centrally presented probe for the N2pc, SPCN, N3rs, and P3. The same time windows and electrodes were selected as in Experiment 1.

Figure 5A presents mean amplitude of the lateral difference waves (contralateral minus ipsilateral) plotted as a function of retention interval. Similar to Experiment 1, the main effect of laterality was significant [F(1, 11) = 6.561, p < .05]. This replicates the result from Experiment 1, demonstrating a significant N2pc component to a centrally presented probe. Interestingly, unlike Experiment 1, laterality and retention interval did not interact significantly (F < 1). The linear decrease in amplitude of the N2pc was not observed in Experiment 2 (see Figure 2 for Experiment 1 and Figure 5 for the pattern in Experiment 2). The sustained N2pc amplitude across retention intervals suggests that the visuospatial representation was not recoded into a new format while the verbal working memory task was used. A repeated measures ANOVA computed on the SPCN data revealed a main effect of Laterality [F(1, 11) = 8.34, p < .01] with ipsilateral (4.12 μV) subtracted from contralateral (3.52 μV) potentials indicating a significant SPCN (−1.40 μV). Similar to Experiment 1, laterality did not interact with retention interval indicating no significant changes in amplitude of the SPCN as a function of retention interval.

Figure 6A plots mean amplitudes of the N3rs component as a function of retention interval separately for three midline electrodes. A repeated measures ANOVA performed on the N3rs revealed a significant main effect of Retention Interval [F(3, 33) = 4.232, p < .01], indicating an increasing negativity as a function of retention interval. The N3rs negativity rose from delay of 300 msec (−0.62 μV), across delay of 700 msec (−0.81 μV) and delay of 1200 msec (−1.75 μV) up to delay of 1700 msec (−1.971 μV). This effectively contrasted with the results from Experiment 1. Given that the N3rs is susceptible to the amount of spatial information represented in memory, it might reflect the persistent spatial coding of VSTM contents in Experiment 2. Having suppressed the possibility of recoding contents into a verbal code, spatial coding seems to have been maintained to serve as recall mechanism even at the longest intervals.

Figure 6C shows behavioral performance in Experiment 2. A repeated measure ANOVA with Greenhouse–Geisser correction for the RT data revealed no significant effect of Retention Interval, although response times differed numerically between conditions. Contrary to the findings in Experiment 1, no linear trend was observed (cf. Figures 4C and 6C). For error rates, a repeated measures ANOVA with Greenhouse–Geisser correction determined a significant main effect of Retention Interval [F(1.841, 23.934) = 8.703, p < .05]. A contrast analysis revealed that delay of 1200 msec resulted in the highest error rate (15%) as compared with 8% (delay of 300 msec), 7% (delay of 700 msec), and 9% (delay of 1700 msec). No sign of linear improvement of performance was observed either in RT or in error rates.

A repeated measures ANOVA on the P3 data revealed a significant main effect of Retention Interval [F(3, 33) = 4.039, p < .01]. Similar to Experiment 1, the positivity increased linearly from delay of 300 msec (1.880 μV) to 700 msec (2.202 μV), 1200 msec (2.939 μV), and 1700 msec (3.137 μV). The main effect of Electrode was also significant [F(1, 11) = 33.651, p < .001], with greater positivity over electrode POz (3.59 μV) than Pz (1.49 μV). The interaction between factors was not significant (see Figure 6B).

In the present study we were able to show a dynamic representational change in memory. Initially objects were stored in visuospatial form. However, as the retention interval increased the spatial form was gradually replaced by a verbal representation. Using electrophysiological measures together with a memory search task and varying the retention interval between memory array presentation and the recall probe stimulus, we were able to study the neural basis for representational recoding in VSTM. We observed an initial representation that is suited to attentional orienting at the shortest retention intervals. However, with an increasing interval, this early representation was gradually recoded into a verbal form, indicating that both attentional orienting to VSTM (N2pc) and spatial retroactive target selection (N3rs) got suppressed with increasing delay. The small (but nonsignificant) N2pc or N3rs at the longest interval in Experiment 1 do not contradict this claim, as we are suggesting a gradual representational change that may not follow the same time course on every trial. Interestingly, behavioral performance as well as the amplitude of the P3 in Experiment 1 showed an inverse pattern (see Figure 4C and B). The P3 has been related to updating of information in working memory (Nobre et al., 2008; Vogel et al., 1998). This updating process has been interpreted as leading to the formation of a more durable, consolidated short-term representation (Vogel et al., 1998). The current increase in P3 with an increased delay suggests that with a longer retention interval updating was also prolonged, which was reflected by an enlarged P3. This prolonged updating process may have allowed for a more consolidated, durable representation at long delays. Interestingly, the increase in P3 amplitude was also observed in Experiment 2, without the improvement in behavioral performance at longer intervals. Given that recoding into verbal code was suppressed in that experiment, the P3 seems to reflect the formation of a more abstract representation, irrespective of the code in use. Together the results from the two experiments led us to conclude that the initial visuospatial memory representation was replaced by a verbal memory representation during the consolidation process.

Alternatively one may explain the increasing amplitude of the P3 component and the decreasing RT observed with longer retention intervals as a consequence of response probability (Brookhuis, Mulder, Mulder, & Gloerich, 1983; Bertelson & Boons, 1960) as the likelihood of having to give a response gets larger as a function of retention interval. As in Experiment 2, the same probabilities were used as in Experiment 1 and no linear decrease in RT was observed, it seems unlikely that probability differences could explain the current result pattern. Modulations of the P3 component, however, may also be attributed to changes in response probabilities over delay.

The attentional orienting hypothesis (Kuo et al., 2009; Lepsien & Nobre, 2007; Griffin & Nobre, 2003) proposes that recall is performed by shifting attention toward spatially organized memorized items. Our results extend the current attentional orienting hypothesis, which does not predict additional VSTM recall mechanisms for spatial information. By providing an electrophysiological measure of the dynamics of representational change in memory recall, the current study speaks in favor of attentional orienting as an initial strategy used only before recoding of information into a verbal code. Using a verbal working memory task, we demonstrated that, when possible, the initial visuospatial representational format was gradually recoded into verbal information as the retention interval increased. Additionally, as revealed by RT data, the benefit from recoding at long retention intervals was not observed when the articulatory suppression task was used. Together this led us to conclude that verbal recoding is associated with a representation that is more efficient in guiding the recall process.

The fact that visuospatial information is getting gradually recoded into verbal form with an increasing retention interval may be linked to concepts of temporal integration. Studies using the temporal integration task (Akyürek, Schubö, & Hommel, 2010; Jiang, 2004) and the attentional blink task (Raymond, Shapiro, & Arnell, 1992) have shown evidence for the existence of a temporal window in which consolidation is expected to happen. Here we provide evidence for consolidation by revealing correlates of the representational change of memory content throughout the consolidation process. We show that the early, preconsolidated representation is spatiotopic and that the spatial layout is still preserved at that time. This spatiality of early representation is gradually recoded into a verbal form over time. This could serve as a mechanism that closes the window of opportunity for temporal integration. Paradigms that are used to study temporal integration often require combining spatially distinct information across successive presentations to form a coherent percept (Akyürek, Schubö, et al., 2010; Jiang, 2004). Here, we show that with increasing retention interval the spatiality of the memorized representation is gradually recoded into verbal form, making spatial information less obvious. This could be one reason why integration is mostly observed at short time delays, when the spatial format is still available (Jiang, 2004). Further consolidation brings up a verbal representation that prevents orienting as a recall strategy. Lacking spatial information, the late form of VSTM representation might also prevent successful temporal integration at long delays (Akyürek, Schubö, et al., 2010; Jiang, 2004). As the spatial information about the initial stimulus array in a sequence is not available anymore, the following array, although being represented in spatial form itself, cannot be combined with the representation of the initial array. Thus, the observed representational change might also provide a limit for temporal integration in VSTM.

The decreasing amplitude of the N2pc and the N3rs could also reflect changes in the way attention operates in VSTM over retention intervals. For example, Zhang and Luck (2009) reported that a representation of feature–location binding fades away earlier than the feature (i.e., color or shape) itself. Participants might still represent the red color of stimuli which they are asked to recall, but the missing feature–location binding prevents them from accessing this information. Thus, participants still might have a representation of a feature after long delays, yet the feature–location bindings are missing, and therefore, they do not know which item they are expected to report. Given the results from Experiment 2 with sustained N2pc and increasing N3rs amplitudes, we claim that locations are still represented in VSTM at long retention intervals. Using the verbal working memory task to increase the difficulty of recoding into a verbal form, we showed that it was possible to use visuospatial attention to orient toward memorized locations even at the longest intervals. Along the same line, Treisman and Zhang (2006) have reported that locations play an important role for maintenance and recall at the initial encoding and that locations and object information are separated in VSTM. The current results extend this view providing evidence that locations could be used to recall information even at longer intervals. This, however, holds only when a more efficient (i.e., verbal) way of representing information in VSTM is prevented.

This research was supported by the German Research Foundation (DFG), by a grant to A. S. as part of the Excellence Cluster “Cognition for Technical Systems” (CoTeSys; Project #433), and by a scholarship to M. L. and N. M. by the Graduate School of Systemic Neurosciences GSN-LMU.

Reprint requests should be sent to Marcin Leszczyński, Department of Experimental Psychology, Ludwig Maximilian University, Leopoldstr. 13, 80802 Munich, Germany, or via e-mail: [email protected].