Set-relevance Determines the Impact of Distractors on Episodic Memory Retrieval

Attention and memory are traditionally studied as separate topics, and the interaction between the two has been relatively neglected. Nonetheless, in most situations, the interplay between these two systems is intimate and two ways (Hutchinson & Turk-Browne, 2012). Memory can serve to modulate the allocation of attention, facilitating perception of current stimuli (Stokes, Atherton, Patai, & Nobre, 2012; Summerfield, Rao, Garside, & Nobre, 2011; Chun & Jiang, 2003), whereas the deployment of attention is necessary for effective memory encoding (Turk-Browne, Golomb, & Chun, 2013; Baddeley, Lewis, Eldridge, & Thomson, 1984), working memory maintenance (Zanto & Gazzaley, 2009), and retrieval (Guerin, Robbins, Gilmore, & Schacter, 2012; Craik, Govoni, Naveh-Benjamin, & Anderson, 1996). Here, we introduce a novel paradigm with the aim of engaging both functions concurrently within the same trial. We presented irrelevant distractors, while participants were asked to retrieve temporal information from episodic memory. By manipulating the temporal and content relationships between the irrelevant sensory distractors and the task-relevant memory probes, we investigated the interplay between the selection/filtering of information in the external world (attention) and the selection of information stored internally (memory retrieval).

Several previous neuroimaging studies highlighted that both attention control and episodic retrieval involve areas in the frontal and parietal cortex (see Naghavi & Nyberg, 2005, for a review). In the memory domain, the parietal cortex has been implicated in a wide range of functions. Among them are its role(s) in successful recollection of source details (Hayama, Vilberg, & Rugg, 2012), as an online “episodic buffer” of recollected information (Vilberg & Rugg, 2008, 2012), and in the accumulation of evidence leading to an eventual mnemonic decision (Wheeler & Buckner, 2003). With respect of the possible relationship between memory and attention, it has been put forward that the parietal cortex may support the directing of attention toward internal mnemonic representations (e.g., Wagner, Shannon, Kahn, & Buckner, 2005). Specifying this relationship further, Ciaramelli, Grady, Levine, Ween, and Moscovitch (2010) demonstrated a dissociation between the dorsal and ventral parietal cortices in episodic memory retrieval, which parallels the classical distinction between endogenous attention control in the intraparietal sulcus and FEFs versus exogenous control in right TPJ (rTPJ) and the right inferior frontal gyrus (rIFG; see Corbetta, Patel, & Shulman, 2008; Corbetta & Shulman, 2002). The finding of such correspondences has led researchers to propose that the orienting to external signals (attention) and to internal/stored information (memory) may involve some common processes (Chun & Johnson, 2011; Cabeza, Ciaramelli, Olson, & Moscovitch, 2008; Ciaramelli, Grady, & Moscovitch, 2008; but see Sestieri, Shulman, & Corbetta, 2010; Hutchinson, Uncapher, & Wagner, 2009).

Together with the parietal cortex, frontal regions have also been implicated both in attention and episodic memory retrieval. The left ventrolateral pFC is activated during retrieval of semantic knowledge (Prince, Tsukiura, & Cabeza, 2007). These effects are thought to reflect the involvement of interference resolution processes (Milham et al., 2001) and/or postretrieval selection to discriminate among competing alternatives. In particular, the IFG appears to support a generalized control process that selects relevant information from among competitors (Badre, Poldrack, Paré-Blagoev, Insler, & Wagner, 2005; Thompson-Schill, D'Esposito, Aguirre, & Farah, 1997). The need to select between alternative possibilities in the memory domain bears some similarity with attention control mechanisms that, together with orienting/shifting operations, are needed to process task-relevant targets in the presence of irrelevant distractors. Examples include visual search paradigms when participants have to identify a target among distractors and selective attention tasks when participants have to maintain attention to a specific location (or feature) while ignoring distractors presented at other locations. These tasks entail endogenous, top–down control processes and activate intraparietal sulcus and FEF bilaterally (Fairhall, Indovina, Driver, & Macaluso, 2009; Hopfinger, Buonocore, & Mangun, 2000; Kastner & Ungerleider, 2000). By contrast, rTPJ and rIFG are activated when a single target is presented at an unattended location (Corbetta & Shulman, 2002) but are deactivated in selective attention tasks that include competing distractors (Shulman et al., 2003; see also DiQuattro & Geng, 2011). Thus, in attention/perception, the “filtering” operation that is required when participants have to maintain a task goal in the face of distracting signals has been associated primarily with deactivation of the rTPJ–IFG network (Corbetta et al., 2008; see also Sestieri, Corbetta, Romani, & Shulman, 2011; Sestieri et al., 2010; Todd, Fougnie, & Marois, 2005).

These attention-related findings appear to be discrepant with the memory results that show increased activation of IFG in the presence of distracting competitors (e.g., Badre et al., 2005). However, recent data may provide us with a framework to reconcile these seemingly contradictory results. Attention tasks that showed deactivation of the ventral frontoparietal network made use of irrelevant stimuli, which do not have any relationship with the task-relevant targets (Shulman et al., 2003; see also Indovina & Macaluso, 2007; Kincade, Abrams, Astafiev, Shulman, & Corbetta, 2005). In fact, studies that specifically manipulated the relationship between the target and the distractors found the target–distractors relatedness to be a key determinant of the response pattern in the ventral attention system. Collectively referred as studies of “contingent attention capture” (Folk, Leber, & Egeth, 2002; Folk, Remington, & Johnston, 1992), several imaging experiments found that distractors sharing some goal-defining feature with the target activate, rather than deactivate, the rTPJ (Natale, Marzi, & Macaluso, 2010; Serences et al., 2005). For example, in a sustained visual task requiring participants to name red letters at one location, the occurrence of a red nonletter distractor at a different location will trigger contingent capture of attention: that is, attention is pulled toward the distractor location, because the attribute “redness” is part of the current task set (Serences et al., 2005; see also Geng & Mangun, 2011). Thus, in attention/perception, the ventral attention system responds to task-irrelevant stimuli only when sharing some aspect with the task-relevant targets (i.e., set-relevant distractors). These paradigms highlight the distinction between “task relevance” and “set relevance.” Task relevance refers to some characteristics of the experimental setting that requires a judgment/response (e.g., a feature of the target) or that provides some information that is potentially useful to perform the task (e.g., a spatial cue). By contrast, “set relevance” refers to irrelevant aspects that do not provide any information to execute the task but are still related to the current task set and, because of this, can influence performance and brain activity.

In standard memory tasks, foils/competitors typically share some feature with the memory probe: for example, belonging to the same study list, the same semantic category or same paired-associate (Guerin et al., 2012; Fletcher, Shallice, Frith, Frackowiak, & Dolan, 1998). This suggests an analogy between selection mechanisms in memory tasks and in attention capture paradigms. Indeed, memory studies of “proactive interference”—which formally manipulate task set during memory retrieval—showed greater activation of ventrolateral frontal areas when participants had to judge nontarget stimuli that were previously/recently used as target (i.e., high set relevance) versus nonrecent trials (Badre & Wagner, 2005; Jonides, Smith, Marshuetz, Koeppe, & Reuter-Lorenz, 1998). Within this framework, the attention–memory correspondence appears to hold, with the ventral frontoparietal system being engaged primarily when the task requires selection between external (attention) or internal (memory) items that compete because they share some aspect that is relevant for the current task set.

Nonetheless, it should be noted that in proactive interference paradigms the nontarget probes are task relevant, that is, participants are required to perform some memory judgment about these stimuli (Badre & Wagner, 2005; Badre et al., 2005; Henson, Shallice, Josephs, & Dolan, 2002; see also Guerin et al., 2012, for an example concerning “shared” semantic category between target and distractors during recognition). This type of paradigm differs substantially from contingent capture paradigms, where participants are told in advance what stimuli to judge (targets) and what stimuli to ignore (exogenous cues, e.g., Folk et al., 1992; or stimuli at an irrelevant location, Serences et al., 2005); and hence, the capture process relates to fully task-irrelevant, albeit set-relevant, distractors (see also above).

A closer parallel to the perceptual situation comes from memory studies that used fully irrelevant distractors on memory retrieval. In a series of experiments, Wais and colleagues reported that task-irrelevant visual distractors reduce retrieval accuracy (Wais, Martin, & Gazzaley, 2012; Wais, Kim, & Gazzaley, 2011; Wais, Rubens, Boccanfuso, & Gazzaley, 2010). In these studies, participants were instructed to direct their full attention to recall the number of items in previously studied images. The retrieval conditions manipulated whether participants' eyes were closed, open and looking at a gray screen, or open and looking at irrelevant visual stimuli. Behavioral data showed diminished recollection performance in conditions when the eyes were open and irrelevant visual stimuli were presented, compared with both gray screen and eyes shut conditions. The imaging data associated the diminished recollection with the disruption of functional connectivity in a network involving the IFG and the inferior parietal lobule (including TPJ). The causal role of the IFG was subsequently confirmed in an rTMS study (Wais et al., 2011). These results demonstrate that irrelevant visual input can interact with memory retrieval in a stimulus-driven manner. However, these paradigms did not tackle the issue of whether/how shared features between memory probes and distractors affect such interactions (cf. set relevance, above).

Here we investigated the interplay between stimulus-driven attention and episodic retrieval by presenting irrelevant distractors during retrieval of temporal information from memory and, specifically, by manipulating the relationship between the distractors and the memory probes. On the basis of a previous paradigm (Kwok, Shallice, & Macaluso, 2012), we used naturalistic stimuli to identify behavioral and fMRI indices associated with the retrieval of temporal information for episodic memory. At encoding, participants were asked to watch a 45-min movie involving complex storylines and dynamic real life-like events, which lasted roughly the length of time of the film. At retrieval, 24 hr later, participants were presented with two images extracted from the film and were asked to report which had occurred earlier at presentation. We previously found that the temporal distance between the probes at encoding affected retrieval performance, with short distances yielding slower retrieval than long distances (Kwok et al., 2012; see also St. Jacques, Rubin, LaBar, & Cabeza, 2008). We attributed the costs of retrieving short instead of long temporal distances to reconstructive search processes (cf. Friedman, 1993) and associated these with activation of the precuneus.

In this study, we modified the retrieval task by presenting the task-irrelevant distractors interleaved between the two probe images (Figure 1A). The distractor pictures were also extracted from the same movie and had specific temporal and semantic relationships with the memory probes (Figure 1B). From a theoretical perspective, existing accounts of temporal order retrieval postulate either a “chronological organization” of the memory traces that emphasizes temporal aspects (Friedman, 1996, 2001) or a reconstructive “knowledge-based” process that instead stresses the role of content (“reconstructive theories”; Friedman, 1993, 2007). Here, we operationalized these two viewpoints through the experimental factors of “D-I content” (content relationship: distractor and probes from same vs. different storylines of the movie) and “D-I proximity” (temporal relationship: distractor extracted from a movie time point close to the first vs. the second memory probe), together with the original factor of temporal distance (“Im1-Im2 distance”: short vs. long distance between the two memory probes).

Depending on whether the distractor taps into the temporal or content dimension that is relevant for the temporal retrieval task (cf. set relevance of the distractors), we would infer which type of processes (i.e., chronological organization vs. reconstructive-based) better characterize the temporal order retrieval. In line with the reconstructive-based account (Kwok et al., 2012), we predicted that the content relationship between the distractors and the memory probes would affect activity in the precuneus more than the temporal relationship (i.e., an effect of “content,” rather than “proximity”).

Furthermore, because of the hypothesized link between attention and memory (cf. above), we anticipated that the type of distractor would affect activity in the ventral attention network (e.g., Ciaramelli et al., 2010; Wais et al., 2010). Specifically, if the factor “D-I content” gave rise to a stronger modulation in the attention network during retrieval, it would suggest that set relevance is content related; whereas if the factor “D-I proximity” resulted in stronger changes, it would suggest that set relevance is temporal related. These distractor factors may further interact with the factor of distance, if the distractor-probes relationship is important only when reconstructive search processes are engaged (i.e., on short-distance trials). The results would extend previous proposals about set-related effects in attention/perception to the domain of retrieval from episodic memory. Moreover, localization of brain activity within attention-related areas during memory retrieval would support recent accounts suggesting a neural overlap between attention selection in perception and in memory (Cabeza, Ciaramelli, & Moscovitch, 2012b; see also Nelson, McDermott, & Petersen, 2012).

The experimental protocol consisted of two phases, encoding and testing, organized across two consecutive days. On Day 1, participants were asked to watch one single 45-min episode of a TV series. On Day 2, during fMRI scanning, participants were tested on a temporal order retrieval task coupled with different manipulations of distraction (i.e., temporal/content relationship between the probes and distracter image). Each retrieval trial included the presentation of a triplet of static images extracted from the film: First Probe Image (Im1), Distractor (D), and Second Probe Image (Im2; see Figure 1A). The participant's task was to choose the probe image that had happened earlier in the film at encoding, ignoring the interleaved distractor image. The experimental design considered the factors of “Im1-Im2 distance” (long/short: long or short temporal distance between the two task-relevant probe images), “D-I content” (same/different: distractor from the same or different storyline as the probe images) and “D-I proximity” (D_im1/D_im2: distractors extracted from a time point closer to either Im1 or Im2; see Figure 1B). In addition, the design included a control condition using scrambled images distractors (Cnt trials), which did not have any relationship with the memory probes.

Fifteen subjects participated in the fMRI experiment (mean age = 22.6 years, 20–30 years; 10 women). None of them had seen the TV episode in question. All had normal or corrected-to-normal visual acuity, and none of them reported neurological impairments. The study was approved by the Fondazione Santa Lucia (Scientific Institute for Research Hospitalization and Health Care) Independent Ethnics Committee, in accordance with the Declaration of Helsinki. A different group of 12 subjects (mean age = 25.4 years, 19–31 years; 10 women) participated in the preliminary behavioral study (see also Figure 2).

The encoding material was one episode of the American TV series “24” (Season 6, Disc 2, 11:00–12:00), which contained five concurrent storylines portraying different characters at various locations (Plot A: depiction of the president and his team in the White House; Plot B: interactions of inmates in a detention center; Plot C: happenings in the office of the Counter Terrorism Unit; Plot D: depiction of Agent Jack on the move; Plot E: a middleman working for the terrorists and his girlfriend). The 45-min episode represents 1 hr of seamlessly timed events; from a temporal perspective, watching it resonates with seeing “real-world” events unfolding over time.

Before encoding (Day 1), participants were instructed to concentrate on the film and memorize as much of it as possible. They were made aware that their memories of the film would be tested on the ensuing day, but no information was provided regarding how and what specific memories would be targeted. Before retrieval (Day 2), participants received detailed task instructions with a screen display example and familiarized themselves with using a keyboard for making responses. Each retrieval trial included the presentation of a triplet of static images extracted from the film (see Figure 1A and section below).

At retrieval, the three images were presented in a sequential order: First Probe Image (Im1, for 1500 msec), Distractor (D, for 500 msec), and Second Probe Image (Im2, for 1500 msec). There was a 1000-msec blank screen between Im1 and D and a 500-msec blank screen between D and Im2 (Figure 1A). The task was to choose the target image that had happened earlier in the film at encoding, ignoring the interleaved Distractor image. On half of the trials, Im1 was the response target; in the other half of the trials, Im2 was the target. Participants were instructed to respond with a keypress as soon as they could make a decision after the second probe image appeared on the screen, but no more than 2.5 sec after it disappeared. Participants indicated the target stimulus (either Im1 or Im2) by pressing one of the two keys with their dominant hand. The stimuli were presented using Cogent Toolbox (www.vislab.ucl.ac.uk/cogent.php) running under Matlab 7.4 (The MathWorks, Natick, MA). A total of 264 trials were presented in four separate fMRI runs. The ITIs were jittered and in the range of 3–5 sec, uniformly distributed.

Memory probes and distractor images were selected based on a film content analysis. The film was segmented into 103 epochs, such that each epoch contained only one scenario/setting. Twenty-one epochs were discarded because they contained unnaturalistic images (e.g., multiple windows/scenes on a single frame), leaving 82 epochs for the image extraction. Each epoch was assigned to one of the five storylines. This identified 26 “n ≥ 3. This ensured that the two probe images (Im1 and Im2) always belonged to the same plot/storyline, yet there was at least one “in-between” cluster. The Im_list included 488 possible couplings. For each epoch E, the two same-storyline distractor lists (SamePre, SamePost) included all the epochs of the previous or next same-storyline cluster, whereas the different-storyline distractor lists (DiffPre, DiffPost) included epochs belonging to a previous/following cluster of a different storyline. This generated the following numbers of probe/distractor epoch couplings: SamePre = 179, SamePost = 132, DiffPre = 306, DiffPost = 307.

For each of the 488 couplings between the two probe images (Im_list), we selected a distractor epoch from one of the four distractor lists or assigned the pair of probes to the control condition. With this, we obtained a set of triplets including two epochs for the probes and one epoch for distractor, plus a set of probes' pairs for the control condition. Then we sampled a subset of 192 triplets ensuring that this final list included 96 triplets with probes and distractor from the same storyline (48 triplets with distractor from SamePre list, 48 with distractor from SamePost list) and 96 triplets with different storyline distractors (48 with DiffPre and 48 with DiffPost distractors). For the control condition, we retained 72 probes' pairs. This constituted the final list of epoch associations that was employed to extract the single frames to be used in the 264 trials.

The single frames (probes and distractors) were extracted randomly from the selected epochs and visually inspected to avoid any unsuitable image (e.g., a frame including a close-up view of a wall). The distractors for the control condition were chosen from a different episode of the same movie series. Each frame was divided in 1170 squares, which were randomly rearranged to generate the scrambled-distractor images. At this stage, we randomly decided which of the two memory probes would be presented first during retrieval (i.e., the Im1). For half of the trials, Im1 was the probe image that occurred earlier in the movie, whereas in the other half the probe that occurred later was used as Im1. The assignment of Im1/Im2 was balanced for same/different storyline, but not for Pre/Post distractors' lists.

We assigned the 264 triplets (cf. above) to the eight conditions given by the crossing of the factors of distance, storyline, and proximity (192 trials), plus the two control conditions (72 trials). Trials were assigned to the “short distance” conditions when the temporal distance between the two probes was shorter than the median distance of all trials (21.2 min); otherwise, the trial was assigned to a “long distance” condition. For the 192 trials belonging to the main 2 × 2 × 2 design, the storyline condition depended on whether the distractor was selected from the “same” or “different” storyline as the two memory probes (cf. Same vs. Diff distractor lists). For the proximity factor, we considered the distance between the distractor and the two memory probes (Im1 and Im2). When the distractor was temporally closer to Im1 than Im2, the trial was assigned to a D_im1 condition, otherwise to a D_im2 condition. The final number of trials for each of the conditions is reported in Figure 2 (range for the eight main conditions = 22–26 trials).

With these procedures, we minimized idiosyncrasy in the stimuli and in the trials assignment to the different conditions. However, we acknowledge that with such complex and naturalistic material we cannot exclude the possibility that conditions differed also with respect of some other unexplored factor.

All images were acquired with a Siemens Allegra (Siemens Medical Systems, Erlangen, Germany) 3T scanner equipped for EPI. A quadrature volume head coil was used for radio-frequency transmission and reception. Head movement was minimized by mild restraint and cushioning. Thirty-two slices of fMRI images were acquired using BOLD imaging (3 × 3 mm in-plane, 2.5 mm thick, 50% distance factor, repetition time = 2.08 sec, echo time = 30 msec, flip angle = 70°, field of view = 192 mm, acquisition order = continuous, ascending), covering the entirety of the cortex.

Data preprocessing was performed with SPM8 (Wellcome Department of Cognitive Neurology, London, United Kingdom) as implemented on MATLAB 7.4. A total of 1548 fMRI volumes for each participant were acquired in four separate fMRI runs. After having discarded the first four volumes in each run, images were realigned to correct for head movements. Slice acquisition delays were corrected using the middle slice as a reference. Images were then normalized to the MNI EPI template, resampled to 3 mm isotropic voxel-size, and spatially smoothed using an isotropic Gaussian kernel of 8 mm FWHM. Time series at each voxel were high-pass filtered at 128 sec and prewhitened by means of autoregressive model AR(1).

The aim of the functional imaging analyses was to identify brain areas showing differential responses as a function of the temporal/content relationship between the distractors and the memory probes. Accordingly, the fMRI analyses considered eight types of trials given by the crossing of the factors of “Im1-Im2 distance” (short/long, temporal distance between the probes), “D-I proximity” (D_im1/D_im2, distractor probe temporal relationship), and “D-I content” (same/different, distractor probe content relationship; see also Figure 1B).

The first-level multiple regression models (i.e., single-subject analyses) included eight main conditions of interest (“Im1-Im2 distance” × “D-I proximity” × “D-I content”), plus two control conditions (scrambled-distractor, with short or long temporal distance between the two probe images). In addition, errors and movement parameters were included as effect of no interest. Each trial was modeled as a short epoch time-locked to the onset of the first probe image (duration = 5 sec, i.e., till the offset of the second probe image), convolved with the canonical hemodynamic response function in SPM8.

The parameter estimates of the conditions of interest were then assessed at the second-level for random effect statistical inference (Penny & Holmes, 2004). The analysis consisted of a within-subject ANOVA considering the 10 conditions described above. To account for any potential difference in task difficulty, condition-specific RTs for each participant were entered as a covariate. Accordingly, here any significant difference between conditions cannot be explained by RT differences (e.g., short temporal distance trials taking longer than long-distance trials; see also Kwok et al., 2012). Moreover, we also estimated a model that did not include the RTs as a covariate and replicated all the main findings (i.e., main effect of “distance” and the “Distance × Proximity” interactions significant after correction for multiple comparisons and “Distance × Content × Proximity” at uncorrected level), again indicating that RT differences between conditions did not influence our results. Correction for nonsphericity was used to account for possible differences in error variance across conditions and any nonindependent error terms for the repeated measures (Friston et al., 2002).

We tested the overall effect of “Im1-Im2 distance” comparing “short vs. long” trials irrespective of distractor condition (cf. St. Jacques et al., 2008), the main effects of distractor “D-I proximity” and “D-I content,” and the interactions between the three factors. With this model, we also compared the eight main conditions versus the low-level baselines (i.e., trials with scrambled distractors). For all comparisons, the statistical threshold was set to p-FWE < .05, whole brain-corrected at cluster level (cluster size estimated at p-unc. = .005).

In a preliminary behavioral study, we found that presenting task-irrelevant distractors during retrieval affected the participants' performance on our temporal order judgment task. ANOVAs with the factors of “Im1-Im2 distance,” “D-I content,” and “D-I proximity” revealed the expected main effect of temporal distance between the two memory probes: participants were faster, F(1, 11) = 45.8, p < .001, and more accurate, F(1, 11) = 27.1, p < .001, on “long” compared with “short” trials (see Kwok et al., 2012; St. Jacques et al., 2008). Both the accuracy data, F(1, 11) = 5.1, p < .046, and RTs, F(1, 11) = 6.9, p < .024, showed an interaction between proximity and distance. The RT data showed a significant three-way interaction between distance, content, and proximity, highlighting the joint influence of both temporal and content relationships between distractors and memory probes on the temporal order judgment, F(1, 11) = 6.4, p < .029. Figure 2 shows that this complex behavioral pattern arose primarily because D_im1 distractors (i.e., those temporally related to the first probe image) speeded up retrieval times compared with D_im2 distractors, specifically when the Im1-Im2 distance was short and the distractor was extracted from a different storyline than the probe images (see red arrows in the panels on the right). In other words, the D_im1 distractors reduced the costs of performing the order judgment on short-distance trials, but did so only when they were not semantically/content related with the memory probes.

The behavioral data collected during fMRI confirmed the main findings of the preliminary pilot study. The main effect of distance was highly significant both in terms of accuracy, F(1, 14) = 17.6, p < .002, and RTs, F(1, 14) = 23.2, p < .001, and the RT data replicated the three-way interaction between distance, content, and proximity, F(1, 14) = 8.9, p < .011. The RT data also showed a Proximity × Storyline interaction, F(1, 14) = 5.6, p < .034, that was not found in the preliminary study. Tukey's HSD post hoc tests on the RTs confirmed that D_im1 distractors speeded up retrieval compared with D_im2 distractors on short-distance, different-storyline trials (p < .017), whereas this difference was not significant for any of the other distance/storyline combinations (all p > .1).

In an additional set of analyses, we also verified whether the correspondence between the order of presentation of the images during encoding and retrieval had any effect on memory performance. First we considered only the task-relevant memory probes (Im1 and Im2), comparing all trials when Im1 was presented before Im2 at encoding (i.e., “congruent order” between encoding and retrieval) versus Im2 before Im1 (i.e., “incongruent order”). In the preliminary experiment, we found slower RTs when Im1 was first (1937 vs. 1866 msec; t(11) = 2.9, p < .015), but this was not fully significant during fMRI (2187 vs. 2138 msec; t(14) = 1.7, p < .122). Moreover, the accuracy data indicated some speed–accuracy trade-off, with greater accuracy on “congruent order” trials (preliminary study: 80.8 vs. 78.6%; fMRI: 80.7 vs. 78.4%), but again the statistical evidence was inconsistent across the two data sets (preliminary study: p < .067; fMRI: p < .229). Next, we considered also the distractor images and compared trials with a “fully congruent image order” (sequence of events at encoding: Im1-D-Im2) versus all the other “incongruent” sequences (i.e., Im2 before Im1, D-Im1-Im2, or Im1-Im2-D). This revealed that participants tended to be slower and more accurate on “congruent” than “incongruent” trials, but again the statistics was not reliable across the two data sets.

In summary, consistently in two independent data sets, the behavioral data showed that (A) participants were slower and less accurate to judge short- compared with long-distance trials and (B) the costs on short-distance trials were reduced by task-irrelevant distractors that—at encoding—were temporally closer to the first than the second probe image (i.e., the D_im1 distractors), especially when the distractors belonged to a different storyline than the two probe images, that is, three-way interaction. These results demonstrate that, although the distractors were fully task irrelevant, some information about this to-be-disregarded stimulus continued to influence participants' evaluation of the Im1/Im2 order (cf. “retrieval inhibition” effects in Bjork & Bjork, 1996). Whether the irrelevant information could affect memory performance depended on both the temporal relationship (“D-I proximity”) and the content relationship (“D-I content”) between the task-irrelevant distractor and the probe images. By the use of functional imaging, we then sought to determine whether this multifaceted behavioral outcome maps onto the activity of one specific brain area/network or can be separately mapped to different areas/networks.

The fMRI analyses considered the overall effect of “Im1-Im2 distance” (short/long), the main effects of “D-I content” (same/different storyline) and “D-I proximity” (D_im1/D_im2), and the interactions between these factors. The main aim of the fMRI analyses was to map the effects of temporal proximity and content relationship between the perceptual distractors and memory probes. Specifically, we asked whether these would include areas where previous studies reported an effect of temporal distance (i.e., the precuneus; see Kwok et al., 2012; St. Jacques et al., 2008) and/or areas previously associated with filtering/selection in stimulus-driven attention and memory (i.e., regions in ventrolateral frontal and parietal cortices; e.g., Cabeza, Ciaramelli, & Moscovitch, 2012a; Ciaramelli et al., 2010; Badre & Wagner, 2007; Corbetta & Shulman, 2002).

First, we sought to replicate our previous findings and confirm greater activation of the precuneus when comparing trials including “short > long” temporal distance between the two probe images (Im1 and Im2). Table 1 lists the regions activated in this contrast. As expected, the main cluster of activation was found in the precuneus (Figure 3, in green), with greater BOLD response for short than long trials, irrespective of distractor conditions (see signal plots in Figure 3). The reverse comparison (long > short) highlighted activation in left anterior superior temporal gyrus (Table 1).

We then turned to test for changes of brain activity as a function of the relationship between the type of distractor and the memory probes. First, we considered the effect of “content” (D-I content) comparing trials including irrelevant distractors extracted from the same versus different storylines. The contrast of “same > different” storyline revealed activation of the precuneus (Figure 3, in red). This was part of a widespread cluster that included adjacent areas in the superior parietal gyrus, but also anterior regions in the frontal cortex. Because of this, we retested the same contrast but using a more stringent voxel-level cluster-defining threshold (p-unc. = .001, rather than .005) that enabled us to better localize the activated areas. The activation of the precuneus and the left superior parietal cortex remained significant, and we localized other activation clusters in the left superior and the left inferior frontal gyri, plus the occipital cortex (see Table 1). The reverse contrast of “different > same” did not reveal any significant activation. Accordingly, the precuneus showed both an effect of the temporal distance between the two task-relevant probes (“Im1-Im2 distance”: short > long) and an effect of the content relationship between the memory probes and the task-irrelevant distractors (“D-I content”: same > different storyline; cf. also Figure 3). Nonetheless, no interaction was found between these two factors. We then also examined the effect of the temporal proximity between the distractors and the memory probes (“D-I proximity”). Contrasts comparing trials including a distractor extracted from a time point close to Im1 vs. Im2 (D_im1 > D_im2) and vice versa (D_im2 > D_im1) did not reveal any significant activation.

We further assessed the influence of irrelevant distractors on temporal order retrieval by testing for the “Proximity × Distance” and the “Content × Distance” interactions. With these we aimed to identify any temporal- or content-related effect that was specific for the retrieval of either short- or long-distance trials, that is, the interactions between distractor-related set relevance and probe-related reconstructive processes that we associated specifically with short-distance trials (cf. Introduction; Kwok et al., 2012).

The contrast testing for the interaction between “proximity” and “distance” (i.e., [D_im1 − D_im2] short > [D_im1 − D_im2] long) revealed three distinct clusters of activation: the rIFG, the rTPJ, plus the left inferior occipital gyrus (see Figure 4A and Table 2). The rTPJ cluster extended ventrally to the STS, and the signal plots of the different peaks within this cluster appeared to show different patterns of activation (note that several conditions contribute to the proximity by distance interaction). We explored this further by comparing separately the simple effects of [D_im1 − D_im2] in “short” trials and [D_im2 − D_im1] in “long” trials. This revealed that the D_im1 distractors in short trials activated the right STS (rSTS), whereas the D_im2 distractors in long trials activated the right supramarginal gyrus (rSMG; see Figure 4B). The reverse contrast ([D_im1 − D_im2] long > [D_im1 − D_im2] short) and interaction between “proximity” and “content” did not give rise to any significant activation.

First, in parallel with the three-way interaction observed in the behavioral data, the highest-order interaction (Distance × Content × Proximity) was associated with changes of activity in the posterior occipital regions. Specifically, the contrast [(D_im1 − D_im2) short > (D_im1 − D_im2) long]_DIFFERENT > [(D_im1 − D_im2) short > (D_im1 − D_im2) long]_SAME revealed activation of the middle occipital gyrus, bilaterally (see Figure 5, in red, and Table 2). These regions showed a complex pattern of activity, including larger activation for “D_im1 vs. D_im2 distractors” in short-distance, different-storyline trials (cf. “bar 3 > bar 4,” signals plots in Figure 5). The reverse contrast [(D_im1 − D_im2) short > (D_im1 − D_im2) long]_SAME > [(D_im1 − D_im2) short > (D_im1 − D_im2) long]_DIFFERENT did not reveal any significant effect.

Second, we assessed whether performance on task covaried with the fMRI responses. For this, we made use of the RT covariate that was initially included in our main statistical model to remove any potential RT confounds (see also Methods section). The effect of the RT covariate revealed a positive covariation between the BOLD signal and RTs in the ventrolateral temporal cortex, bilaterally (right hemisphere: x y z = 74 −36 −10, Z = 5.17, p-FWE-corr. < .001; left hemisphere: x y z = −57 −21 −22, Z = 4.37, p-FWE-corr. < .049). In these regions, activity increased with increasing RTs suggesting a possible link between these areas and performance on task. Because these areas did not show any condition-specific effect, we will focus our discussion primarily on the precuneus and the right ventral attention system that, we suggest, jointly contribute to the final behavioral outcome on task.

We investigated the interplay between stimulus-driven attention and memory retrieval by presenting task-irrelevant visual distractors during retrieval of temporal information about complex events. Behaviorally, trials where there was only a short temporal distance between the two memory probes incurred a retrieval cost on the order judgment task (lower accuracy and longer RTs). The RT costs were modulated by both temporal and content relationships between the irrelevant distractors and the memory probes. Specifically, when distractors were closer in time to the first than the second probe image (i.e., the D_im1 distractors), these costs were significantly reduced. This effect of “distraction” on retrieval performance took place only when the irrelevant distractors were semantically less related to the task-relevant probes (i.e., different storyline trials).

The fMRI results revealed distinct effects of temporal versus content relationship between the distractors and the memory probes. The temporal relationship was found to affect activity in the right ventral attention network (rTPJ-rIFG), dependent on both how temporally close the distractor was to the first probe image and the temporal separation between the two probe images. Within this system, the rSTS was found to be activated when the distractor was close to the first probe image (i.e., D_im1 > D_im2) in short-distance trials, whereas the rSMG showed an effect of D_im2 > D_im1 in long trials.

By contrast, the content relationship between distractors and memory probes was found to affect activity in the precuneus. This region showed increased activation when the trials included distractors extracted from the same storyline as the probe images and also showed an overall effect of temporal distance, with greater activation for trials with short compared with long distances between the two memory probes. Finally, temporal and content relationships jointly modulated activity in the lateral occipital cortex, where the pattern of BOLD signal exhibited some correspondence with the RT data.

These results demonstrate that task-irrelevant distractors can influence memory retrieval and that this crucially depends on both the temporal and content relationships between the distractors and the memory probes, thus highlighting the relevance of the current task set. The interaction between perceptual distractors and memory probes in rTPJ and rIFG suggests a link between selection/filtering in the attention and memory domains, whereas the response pattern of the precuneus implicates this region in memory search within knowledge/content-based hierarchies (cf. “reconstructive theories,” Friedman, 1993, 2007). We discuss these points below.

Linking our current paradigm—which crossed perceptual (distractors) and memory (temporal retrieval) factors—to previous attention research, we predicted that specific probe–distractor combinations would lead to a contingent capture of attention on a memory representation level (see also Ciaramelli et al., 2010; Wagner et al., 2005). Our behavioral measures indicate that the temporal- and content-related relationship between distractors and memory probes affected the retrieval performance in a complex manner (cf. a three-way interaction). Specifically, on seeing the first probe image (Im1), we assume that participants lodged themselves in a mode of retrieving information from a specific time point within their memory of the encoded stimuli (i.e., the top–down goal). In D_im1 trials, the temporal closeness of the distractor to Im1 generated a contingency between the current top–down goal and the task-irrelevant distractor, that is, the time point of presentation of the distractor during encoding matching the currently relevant Im1 time point. We suggest that this correspondence helps to capture or anchor the participant's attention toward that point in time in the memory representation (i.e., the “Im1-D” moment in the movie). This would in turn facilitate the temporal order judgment when the forthcoming second probe image (Im2) is near—at presentation—this “Im1-D” anchor (i.e., the short-distance trials). By virtue of these temporal relationships, attention capture by the D_im1 distractors led to faster responses in short-distance trials.

A somewhat related account of this temporal effect concerns the classical memory finding that the recall of an item is facilitated by the presentation of another item that occurred close in time during encoding (temporal congruity effect). Howard and Kahana (2002) proposed a temporal context model postulating that the encoding of new items dynamically changes the representation of the current temporal context. When an item is retrieved at test, this reinstates the temporal context associated with that item during encoding. As the temporal context will include several neighboring items, the contextual model accounts for the temporal contiguity effect (Sederberg, Howard, & Kahana, 2008; Howard & Kahana, 2002). In the current paradigm, the presentation of the first probe image (Im1) would reactivate a specific temporal context associated with the encoding phase. In D_im1 trials, the distractor would coincide with the reactivated context, which would further strengthen the reactivation and give rise to the improved retrieval performance.

In accord with the attention capture interpretation, the fMRI analyses found that the D_im1 versus D_im2 distractor conditions modulated activity in the right ventral attention network (rTPJ-rIFG). Our main analysis showed a Distance × Proximity interaction, which was then further differentiated into an effect of “D_im1 > D_im2 on short trials” in rSTS and of “D_im2 > D_im1 on long trials” in rSMG (see Figure 3B). Our finding of an interaction between “D-I proximity” and “Im1-Im2 distance,” rather than just a main effect of proximity (D_im1 > D_im2 or D_im2 > D_im1) indicates that activity in the ventral attention system depends both on the relationship between the distractor (D) and the memory probe (Im1), as well as the temporal distance between the two memory probes. In visuospatial attention research, contingent capture paradigms showed activation of the rTPJ when a task-relevant target is presented at a different position compared with the task-irrelevant but “set-relevant” cue (i.e., spatially invalid trials; Natale et al., 2010). The interpretation of this is that the set-relevant cue captures attention, but that rTPJ is activated because additional operations are required during target processing (e.g., spatial reorienting). In our current account (cf. above), the distractor would correspond to the task-irrelevant “cue,” and on D_im1 trials, its temporal proximity to Im1 would correspond to the “feature” that makes that distractor set relevant. Im2 would correspond to the task-relevant target that triggers activation of the ventral attention system, specifically when it followed a D-Im1 pair that captured/anchored attention.

As noted above, separate tests on “short” and “long” distance trials showed that, in opposite fashion to the rSTS, the rSMG was activated more for D_im2 compared with D_im1 distractors, specifically in long-distance trials. This could be interpreted within a framework “reorienting within memory representations” (Cabeza et al., 2012a) with the right SMG being engaged because the temporal association between the distractor and the second probe (D_im2) triggered a shift of attention away (and more so on long-distance trials) from the temporal position of the first probe. This is different from our main account regarding the D_im1 distractors, but in this case the effect of proximity would also depend on both the temporal relationship between the distractor and the probe, as well as the temporal distance between the two probes. With respect to this account, both the rSTS and rSMG were found to process the task-irrelevant distractors as a function of their temporal relationship with the probes (i.e., according to their set relevance).

Here we wish to emphasize that the effects of proximity (main interaction term, Figure 4A, and the two simple effects on short and long trials, Figure 4B) were all found in ventral regions of the right hemisphere. This localization appears consistent with the bottom–up attention hypothesis (Cabeza et al., 2012a) and attention-to-memory models (e.g., Cabeza et al., 2008). Our findings help to extend these proposals as they suggest that the ventral attention system can capitalize on external, task-irrelevant signals to modulate memory temporal order retrieval. We nonetheless acknowledge that our account of set-dependent contingent capture in the memory domain is still somewhat speculative and that additional work, possibly including additional within-participant attention control tasks, could provide more specific evidence about the role of different subregions around the rTPJ (e.g., see Guerin et al., 2012; Sestieri et al., 2010; Hutchinson et al., 2009).¹

Our contingent capture account for the “D_im1 short-distance trials” also has similarities to the notion of “retrieval specifications,” whereby external cues (here, the “Im1-D” anchor) can provide the participant with a memory search “description” (Fletcher et al., 1998; Burgess & Shallice, 1996). Related to this, Fletcher, Shallice, and Dolan (2000) proposed a mechanism of retrieval from memory involving Frith's (2000) concept of the “sculpting” of the response space, with retrieval cues modulating lower-level processes to help to ensure that task-appropriate responses are selected. Both accounts have been associated with activation of the IFG. More broadly, the strong temporal relationship between the distractor and the two memory probes on D_im1 short-distance trials (i.e., all three events were temporally close during encoding) may trigger specific postretrieval selective processes, again a function previously associated with the IFG (Badre & Wagner, 2007; Badre et al., 2005; Thompson-Schill et al., 2002; Fletcher et al., 2000). Nonetheless, in the current framework, the IFG would not be merely implicated in selecting suitable alternatives (Badre et al., 2005) or suppressing unwanted contents (Levy & Anderson, 2012; Depue, Curran, & Banich, 2007), but rather, it would make use of potentially distracting task-irrelevant stimuli to help access relevant information in episodic memory (i.e., the Im1-Im2 temporal order).

In addition to these time-related effects (Distance × Proximity interaction), both behavioral and imaging results revealed an influence of the content relationship between the distractors and the memory probes. Behaviorally, we found that the influence of D_im1 distractors on short trials took place only when they belonged to a different storyline compared with the memory probes (i.e., Distance × Proximity × Content interaction). Notably, at the neurophysiological level D-I content and D-I proximity effects were largely dissociated. Content did not modulate activities in the right ventral attention network but instead modulated responses in the medial parietal cortex including the precuneus. In this area, we found an overlap between the main effect of D-I content, with greater activation for “same” than “different” storylines, and the main effect of Im1-Im2 distance (“short > long”).

We predicted that the precuneus would be involved in reconstructive-based retrieval processes. Confirming this hypothesis and replicating previous studies that also used temporal judgment tasks with naturalistic material (Kwok et al., 2012; St. Jacques et al., 2008), we found modulation of activity in the precuneus as a function of the temporal distance between the two probe images. Activity in the precuneus increased with decreasing temporal distance. Previous studies interpreted this effect, suggesting that the retrieval of temporal events involves search processes within knowledge hierarchies organized by ecologically meaningful grains (Kwok et al., 2012) and/or effortful reconstruction processes (St. Jacques et al., 2008). Here, we found that the content relationship between memory probes and the distractors modulated activity in the precuneus, showing that even a brief exposure to semantically more related images during retrieval affected brain activity in the medial parietal cortex. This highlights the role of the precuneus in dealing with meaningful, related events during retrieval, lending stronger support to content-based reconstructive theories (Friedman, 1993, 2004) than chronological search processes (Friedman, 1996, 2001). Moreover, our current results regarding the precuneus support proposals that implicate this region in parsing streams of content-related complex events into meaningful units during perception (Kurby & Zacks, 2008; Zacks, Tversky, & Iyer, 2001) and structuring these into knowledge-based memory hierarchies (Zacks, Speer, Swallow, & Maley, 2010; Speer, Zacks, & Reynolds, 2007). Overall, these findings contribute to a greater specification of the role of the medial parietal cortex during the retrieval of temporal information about complex memories and highlight the central role of content/semantics rather than mere temporal aspects.

The association of the precuneus with effortful memory search within knowledge hierarchies also fits with current hypotheses about the role of top–down (voluntary) attention during memory retrieval (AtoM: attention-to-memory; Cabeza et al., 2008; Ciaramelli et al., 2008). These theories postulate an attention-related involvement of dorsal parietal regions—including the precuneus—during retrieval search, monitoring, and verification (Cabeza et al., 2008; Ciaramelli et al., 2008). Nonetheless, to our knowledge, these previous models have not yet put forward a detailed characterization mapping specific retrieval operations to subregions within the dorsal parietal cortex. It is therefore possible that different subsets of attention processes are utilized by different top–down memory retrieval operations/tasks. The empirical evidence presented here suggests that the precuneus may be preferentially engaged for operations acting on discriminating the content relationship between episodic events (cf. “reconstructive theories,” Friedman, 1993, 2007).

Together with the precuneus, the content relatedness of the distractor with respect to the memory probes (same vs. different storyline) also showed increased activation in the left pFC. This prefrontal activation is consistent with previous evidence showing that memory for temporal order is dependent on the pFC (Konishi et al., 2002; Suzuki et al., 2002; see also Moscovitch, 1992; Milner, Corsi, & Leonard, 1991, for related work in patients). These frontal effects may be of supervisory/monitoring nature acting upon some “organizational” structure (Shallice & Burgess, 1996). In the context of the current paradigm, we suggest that the same-storyline trials required additional supervisory/monitoring processes to correctly sort the three images (Im1/D/Im2) in relevant memory probes versus the to-be-ignored distractor. This would occur because the high-level of content association between the probes and the distractor in these trials (compared with the different-storyline trials) generated an additional potential source of confusion for the retrieval of the temporal order of the two probes.

Finally, the content relationship between the distractors and the memory probes was found to modulate activity in the visual cortex. In the lateral occipital cortex, we found greater activity for D_im1 than D_im2 distractors specifically in short-distance, different-storylines trials. Thus, in the visual cortex, the pattern of brain activity paralleled the interaction between distance, content, and proximity found at the behavioral level. This result was unexpected, and we can only speculate about the possible underlying processes. On each trial, the task required the participant to suppress the processing of the interleaved distractor image, focusing selectively on the first and third images for the memory judgment. The behavioral advantage for trials including D_im1 distractors indicates that the suppressed representation of the task-irrelevant distractor was reduced (cf. also “directed forgetting,” Bjork & Bjork, 1996). One possibility is that the modulation of activity in visual cortex reflects some aspect of this recovery process, the strength of which may depend both on sensory details of the three pictures presented sequentially (Im1-D-Im2: “repetition priming”; Buckner et al., 1998) as well as inhibitory mechanisms related to the filtering/selection requirements of the current memory task (“negative priming”; see Nee & Jonides, 2008).

Despite the similarity between the pattern of activation in occipital cortex and the behavioral data (three-way interaction), we wish to emphasize that the behavioral performance represents only the final output of multiple operations/processes, likely to be carried out by different areas of the brain. In particular, the performance for the “Dim1/short-distance/different-storyline” trials ought to depend on processes carried out by both the rTPJ-IFG network (cf. our contingent capture account) and the medial parietal cortex, which were activated when participants have to judge on short-distance trials (see also Kwok et al., 2012) as well as being differentially involved when the distractor and the memory probes belonged to the same versus different storyline.

We conclude that bottom–up attention and memory retrieval entail common selection processes and that this involves a multifaceted mechanism. The degree to which task-irrelevant distractor stimuli presented during retrieval affected memory performance, depended on the specific relationship between the irrelevant perceptual signals and the memory information that participants were asked to retrieve (i.e., the task set). Both temporal- and content-related factors were found to play a role. Behaviorally, this led to a complex pattern of retrieval performance, where task-irrelevant distractors that were close in time—at presentation—to the first memory probe reduced the RT costs of retrieving temporal information on short-distance trials. We link this effect of temporal proximity to set-related contingent capture of attention on a memory representation level and associate this with the activation of the rTPJ-IFG network. The content relationship between the distractor and memory probes also influenced the retrieval performance, but in a different way from the temporal effects at the neurophysiological level. The influence of the distractor probe content relationship was found to overlap with the effect of temporal distance between the two probes in the precuneus. This is consistent with the proposal that the involvement of the medial parietal cortex during memory retrieval reflects search processes within knowledge/content-based memory hierarchies.

This work, conducted in The Neuroimaging Laboratory, Santa Lucia Foundation, was supported by The Italian Ministry of Health. This work has also received funding from the European Research Council under the European Union's Seventh Framework Programme (FP7/2007-2013)/ERC Grant Agreement n.242809. The authors thank Paolo Alessandrini, Steve Gazzitano, Fabio Mollo, and Domenico Itri for their assistance.

Reprint requests should be sent to Sze Chai Kwok, Neuroimaging Laboratory, Santa Lucia Foundation, Via Ardeatina 306, 00179 Rome, Italy, or via e-mail: [email protected].