Memory can profoundly influence new learning, presumably because memory optimizes exploration of to-be-learned material. Although hippocampus and frontoparietal networks have been implicated in memory-guided exploration, their specific and interactive roles have not been identified. We examined eye movements during fMRI scanning to identify neural correlates of the influences of memory retrieval on exploration and learning. After retrieval of one object in a multiobject array, viewing was strategically directed away from the retrieved object toward nonretrieved objects, such that exploration was directed toward to-be-learned content. Retrieved objects later served as optimal reminder cues, indicating that exploration caused memory to become structured around the retrieved content. Hippocampal activity was associated with memory retrieval, whereas frontoparietal activity varied with strategic viewing patterns deployed after retrieval, thus providing spatiotemporal dissociation of memory retrieval from memory-guided learning strategies. Time-lagged fMRI connectivity analyses indicated that hippocampal activity predicted frontoparietal activity to a greater extent for a condition in which retrieval guided exploration occurred than for a passive control condition in which exploration was not influenced by retrieval. This demonstrates network-level interaction effects specific to influences of memory on strategic exploration. These findings show how memory guides behavior during learning and demonstrate distinct yet interactive hippocampal–frontoparietal roles in implementing strategic exploration behaviors that determine the fate of evolving memory representations.
Memory retrieval is crucial for adaptive processes such as exploration (Voss, Gonsalves, Federmeier, Tranel, & Cohen, 2011; Voss, Warren, et al., 2011; Okeefe & Nadel, 1979), decision-making (Redish, 2016; Wang, Cohen, & Voss, 2015; Gupta et al., 2009; Johnson & Redish, 2007), future planning (Schacter et al., 2012; Buckner, 2010; Eichenbaum & Fortin, 2009), creative thinking (Duff, Kurczek, Rubin, Cohen, & Tranel, 2013), and memory updating (Bridge & Voss, 2014a, 2014b; Bridge & Paller, 2012; Iordanova, Good, & Honey, 2011; Winters, Tucci, Jacklin, Reid, & Newsome, 2011). However, mechanisms supporting memory's influence on adaptive functions such as learning remain unresolved. The hippocampus is strongly implicated in memory-mediated learning (Bridge & Voss, 2014b; Zeithamova, Dominick, & Preston, 2012; Iordanova et al., 2011; Zeithamova & Preston, 2010; Shohamy & Wagner, 2008) and memory-guided exploration (Redish, 2016; Wang et al., 2015; Wang & Voss, 2014; Kaplan et al., 2012; Buckner, 2010). Both processes involve using memory to determine what information is attended to and therefore integrated into coherent memory representations. Previous studies have shown that memory-guided exploration involves self-directed encoding strategies that can enhance later memory. In these circumstances, memory for previously sampled information presumably is used to guide exploration of new information in a way that promotes memory for the entire episode. This memory-guided exploration process has been hypothesized to involve interactions of hippocampus and frontoparietal networks, which are thought to support memory versus exploration, respectively (Redish, 2016; Wang et al., 2015; Wang & Voss, 2014; Voss, Galvan, & Gonsalves, 2011; Voss, Gonsalves, et al., 2011; Voss, Warren, et al., 2011). However, hippocampal versus frontoparietal contributions to memory versus exploration-related processes triggered by memory have not been fully segregated, as they tend to occur in rapid succession during memory-guided exploration in most circumstances. Thus, although it is known that memory retrieval optimizes exploratory behavior and learning, specific mechanisms are unclear.
Most mechanistic evidence for memory influences on behavior is relevant to how memory dictates choice behavior for learned information (Brown et al., 2016; Redish, 2016; Rich & Wallis, 2016; Wang et al., 2015; Wang & Voss, 2014; Johnson & Redish, 2007). For instance, after learning a T-maze with two competing paths, rats look back and forth between the paths at the intersection/decision point. During this iterative sampling behavior, reactivation of hippocampal place cells associated with each path was identified (Johnson & Redish, 2007). This suggests that the two choices cued retrieval of each path, allowing for simulation of possible outcomes to optimize decision-making. Neural findings have converged on the notion that hippocampus interacts with cortical regions to optimize choice behavior. Given two choices, hippocampal-mediated retrieval of each associated outcome allows for cortically mediated behavioral selection of the most advantageous choice, therefore putting choice behavior in alignment with learning history (Redish, 2016; Wang et al., 2015). Although these studies have focused on the influence of memory for established outcomes and have implicated hippocampal–cortical interactivity in this process, mechanisms for how memory-guided behavior supports new learning remain hypothetical (Redish, 2016; Wang et al., 2015; Tolman, 1932). Rather than simply providing concrete information about past experiences to influence choice between well-learned alternatives, we propose that memory shapes learning by determining how information in the environment is sampled and incorporated into existing representations.
We previously found evidence implicating hippocampal–frontoparietal interactions in memory-guided exploration during new learning (Wang & Voss, 2014; Voss, Galvan, et al., 2011; Voss, Gonsalves, et al., 2011; Voss, Warren, et al., 2011), but we were unable to distinguish hippocampal from frontoparietal contributions to memory retrieval versus memory-guided exploration. Indeed, studies that implicate hippocampal and frontoparietal networks in future-oriented functions such as prediction, prospection, and future imagining have not clearly segregated the role of hippocampus in storing and retrieving memories versus using this information to construct mental images of future possibilities (e.g., Redish, 2016; Schacter et al., 2012; Buckner, 2010; Gupta et al., 2009). Here, we attempted to segregate memory retrieval from adaptive functions that strategically utilize memory to guide behavior. We implemented a novel experimental design that was capable of distinguishing retrieval from memory-triggered exploration processes. We used a combination of behavior and eye-tracking measures to examine neural correlates of memory retrieval and memory-guided exploration as they unfolded over time, which enabled us to pinpoint influences of retrieval on visual exploration and learning.
We hypothesized that memory retrieval would be associated with hippocampal activity and would influence subsequent viewing behavior during exploration (Bridge & Voss, 2015). Because memory retrieval should cause retrieved information to temporarily enter an active state (Lewis, 1979), we reasoned that viewing would be strategically allocated to other, nonretrieved information immediately after retrieval. We predicted that these strategic viewing behaviors during exploration would be related to frontoparietal networks implicated in coordinating planned action sequences (Duncan, 2013). This hypothesized fMRI pattern would provide spatiotemporal dissociation of activity related to memory retrieval from activity related to subsequently deployed exploration strategies influenced by memory. We also predicted that memory-guided exploration would influence the structure of later memories. We hypothesized that retrieved information would become disproportionately bound with the explored information, because more opportunities existed for binding it to explored information, as it remains active in memory while other information is sampled iteratively. This hypothesis provides a specific mechanism for previously reported beneficial effects of memory-guided exploration on subsequent memory (Redish, 2016; Wang et al., 2014, 2015; Wang & Voss, 2014; Voss, Gonsalves, et al., 2011; Voss, Warren, et al., 2011), that is, better integration of retrieved information with explored information.
We used an episodic memory task in which each episode included three objects appearing at arbitrarily assigned locations on a rectangular grid (Figure 1). Each study trial was composed of three periods: initial study, manipulation, and restudy. The manipulation portion was either “active,” involving spatial recall of a selected object, or “passive,” involving reexposure to the selected object location, but no recall demand. We hypothesized that retrieval in the active condition would cause the retrieved object to be preferentially bound with other information during restudy and that viewing behavior would reflect this preferential binding. To assess this hypothesized preferential binding, two types of retrieval cues were used during a subsequent test phase (Figure 1). The retrieval cue was either a manipulated or nonmanipulated (i.e., “other”) object, and participants were prompted to select the associated other object and recall its grid location. This test therefore probed influences of memory retrieval on the resulting structure by determining which content was most central to the resulting memory representation (retrieved content vs. other content).
Data were collected from 23 people. Two participants were excluded from all analyses because performance on the short-term recall task during the active manipulation fell 2 SDs below the group mean (M = 0.50, SD = 0.05). One participant was excluded from the fMRI analysis only because of physical removal from the scanner mid-experiment. Data from 21 participants remained for behavioral analysis (12 women; ages 21–33 years, M = 26.33 years; all right-handed). All participants reported no history of neurological or psychiatric conditions and no current use of any psychoactive drugs. Written informed consent was obtained from all participants before participation in accordance with the Northwestern University institutional review board. Participants were paid for their participation.
Visual stimuli were displayed on an MRI-compatible projector screen with an eye-to-screen distance of approximately 70 cm. Two sets of 216 object images were used, for a total of 432 unique object images. Because of the large quantity of images needed for the experiment, two object sets were used. One set was composed of images of real-life objects (Brodeur, Dionne-Dostie, Montreuil, & Lepage, 2010), and the other set was composed of color drawings of objects (Rossion & Pourtois, 2004). Each block contained objects from only one object set. A white box with dimensions of 3.30 × 3.30 cm encapsulated each object. Most of the screen was occupied by a green rectangular grid composed of 18 individual spaces separated by thick, black lines. Additional gray space at the bottom of the screen was reserved for the object recognition and spatial recall tests. Each object could appear in the center of any of the 18 grid spaces. The grid dimensions were 26 × 13 cm.
Each block was composed of a “study and test” phase, separated by a 1-min distractor task. There were six blocks with 24 study trials and 20 test trials. Each study phase was intermixed with three trial types: 10 active manipulation trials, 10 passive manipulation trials, and 4 catch trials.
Each active/passive study trial had three parts: initial study, manipulation, and restudy. A 1- to 2-sec jittered grid image preceded each study trial. During initial study, participants first viewed three objects in distinct locations on the grid for 4000 msec (Figure 1). There was a delay between initial study and the manipulation phase of the trial, during which participants completed a distractor task. For the distractor task, two fractal images appeared consecutively at the center of the screen, and participants were prompted to indicate if the images were the same or different. The duration of the distractor task was, on average, 6.85 sec (SD = 0.12 sec). Importantly, participants did not know which type of manipulation would occur after the distractor until a cue appeared. Therefore, participants were unaware which type of manipulation would occur during the initial study phase.
At the end of the distractor task, a cue trial indicating the upcoming manipulation type (active, passive, or catch) appeared for 1–2.5 sec. For the active/passive trials, participants next completed the manipulation task. For this manipulation, all three objects were positioned at the bottom of the screen below the grid, and participants were prompted to move one object to its associated location while the other two objects remained at the bottom of the screen. For the active manipulation, participants selected one object (M = 1.60 sec, SD = 0.27 sec) and then recalled its associated grid location by moving the object with a mouse (M = 1.32 sec, SD = 0.38 sec). If the object was placed in the incorrect location, it immediately moved to the correct location. Accuracy was high (M = 84.49% correct, SD = 9.89%), and incorrect trials were excluded from all analyses. For the passive manipulation, participants were prompted to select the highlighted object (M = 1.51 sec, SD = 0.24 sec) and then move it to its associated location on the grid (M = 1.37 sec, SD = 0.45 sec). The preselected object was highlighted by a thick red outline, and its associated location was marked by a red box on the grid. In the passive condition, participants occasionally failed to select the preselected object and move it accordingly within the allotted time limit (5 sec). These trials were omitted from analyses, leaving 98.56% of trials for analysis (SD = 2.00%).
Immediately after placement of the manipulated object, participants restudied the three objects in their locations 4000 msec (restudy). Eye movements were recorded during the initial study and restudy phases of each trial. The Active manipulation included both active selection of an object and active recall of the object's associated location. Therefore, it is possible that both the active selection and retrieval processes contributed to the entrance of the manipulated object into the active memory state. However, we have shown in a prior study that retrieval in the absence of selection influences subsequent memory similarly to the effects reported here (Bridge & Voss, 2015).
In addition to the active/passive trials, four randomly distributed catch trials occurred during the study phase. The catch trials served to encourage participants to view all objects during the initial study period. On catch trials, participants were required to recall one randomly selected object's location after the initial study period. Participants then received immediate feedback on their recall accuracy. Catch trials were not subsequently tested or analyzed.
After the study phase, participants viewed various images of cats (Zhang, Sun, & Tang, 2011) for 1 min as a distractor before the test. After the distractor, the corresponding test was administered. Participants were given an associative object-recognition test and spatial recall test for the nonmanipulated (tested) objects. During each test trial, a fixation cross flashed on and off the screen in the upcoming cue object's location (200 msec on, 100 msec off, 1000 msec on) to orient the participant's attention to that location. Then, the cue object appeared in that location for 2000 msec. Participants attempted to recall the entire episode associated with the cue during this time (including the two other objects and their locations). The cue disappeared, and the grid remained on the screen for 1–3 sec. Then, the cue reappeared along with two object choices at the bottom of the screen. One of the objects at the bottom of the screen was associated with the cue object, and the other object was associated with a different set of studied objects. Each tested object appeared on two test trials: once as the correct associate and once as a familiar lure. Importantly, each object choice appeared in only one cue condition (i.e., if an object was the correct associate of a manipulated cue object, then it also appeared as a lure for a different manipulated cue object on another trial). Participants chose the object that was associated with the reminder cue by clicking on the object with the mouse (M = 1.91 sec, SD = 0.25 sec). Then, participants used the mouse to move the selected object to its associated location on the grid (spatial recall) and clicked a button on the mouse (M = 1.48 sec, SD = 0.37 sec). The objects remained on the screen for 500 msec after the mouse was clicked. A confidence trial followed the spatial test, in which participants were asked to indicate whether the combination of their associative recognition and spatial recall responses were made with low, medium, or high confidence. The trial ended after the confidence decision was made or after 3 sec, whichever came first.
We examined memory for the original multiobject spatial event (Figure 2A) by conducting behavioral analyses of object recognition and spatial recall accuracy. For object recognition, we measured the proportion of trials in which the correct object was selected (Figure 2B). After correct selection of the associated object, we measured spatial recall. We conducted analyses involving memory for the precise object-location pairings (“bull's-eye memory”) and memory for the overall spatial configuration of the objects (“configural memory”). We measured bull's-eye memory by calculating the proportion of correctly recognized objects that were placed in their associated locations on the grid (Figure 2C). We measured configural memory by examining the proportion of correctly recognized objects that were placed in either their associated grid locations or the nontested object's location (i.e., participants remember the spatial layout of the episode, but not necessarily the specific object–location pairings; Figure 2D). Configural memory accuracy was used to subdivide fMRI and eye movement contrasts based on subsequent memory. We intentionally did not test memory for the manipulated object itself, as this was not the focus of the current study. However, we have demonstrated a long-term memory advantage for retrieved information in other studies using similar memory paradigms (Bridge & Voss, 2014b; Bridge & Paller, 2012).
Eye movements were recorded at 500 Hz during the initial study and restudy phases of each trial using an Eyelink 1000 remote tracking system (SR Research, Ontario, Canada) that was focused on the right eye via the mirror that participants used to view the projection screen in the MRI scanner. The continuous eye movement records were transformed into a time series of fixations, saccades, and blinks. Motion (0.15°), velocity (30°/sec), and acceleration (8000°/sec2) thresholds were used to identify saccades. Periods during which the pupil size was very small were classified as blinks. Saccade- and blink-free periods were categorized as fixation events. The average gaze position was calculated over the duration of each fixation event. The duration and time course of fixations in ROIs were analyzed using custom scripts in MATLAB (The MathWorks, Inc., Natick, MA). ROIs were circles that encompassed the three individual objects in their corresponding locations. The average ROI radius was 4.64 cm (SD = 0.78 cm).
Eye movements were recorded during initial study (4000 msec) and restudy (4000 msec). Participants spent an average of 2.79 sec (SE = 0.067) viewing any of the three objects during the initial study phase for the active condition and 2.76 sec (SE = 0.07) viewing any objects in the passive condition (overall viewing time did not differ significantly for these conditions [t(20) = 0.97, p = .34]). For the restudy phase, participants spent an average of 3.05 sec (SE = 0.06) viewing any of the three objects in the active condition and 3.04 sec (SE = 0.05) viewing any of the objects in the passive condition (overall viewing time did not differ significantly for these conditions [t(20) = 0.32, p = .75]). Therefore, overall attention to the three objects did not differ across conditions during either the initial study or restudy phase.
Our primary analysis focused on eye movements after the active/passive manipulation during the restudy period. We analyzed eye movements separately for each restudy trial by first summing the total time spent fixating within the three ROIs (Hannula et al., 2010). We divided the total viewing time for each individual ROI by the total ROI viewing time to obtain a location-based proportion of viewing time measurement for each latency interval, using standard bin sizes (Hannula, Baym, Warren, & Cohen, 2012). Specifically, we examined the proportion of time spent viewing each ROI during eight 500-msec intervals across the entire restudy period: 1–500, 501–1000, 1001–1500, 1501–2000, 2001–2500, 2501–3000, 3001–3500, and 3501–4000 msec. For the eye movement analysis, we report only significant differences with regard to manipulation condition and viewing period. Greenhouse–Geisser correction is denoted by GG in text. Trials with less than 20% total viewing time directed to the three ROIs during the restudy phase were excluded from all eye-movement related analyses. Participants who did not have more than five trials in any condition of interest were excluded from the eye movement analysis that included that condition (see details for each analysis, including trial counts and number of participants contributing to each analysis below).
We performed three different eye movement analyses. First, we examined proportion of time viewing the manipulated object during the restudy phase for the active and passive conditions (Figure 3A). For this comparison, there was a mean of 50.71 trials (SD = 10.20 trials) in the active condition and a mean of 58.19 trials (SE = 7.65) in the passive condition. Data from 21 participants were included in this analysis. For the second and third analyses, we examined the proportion of object-to-object transitions (Figure 3B–C). Trials in which there was at least one object-to-object transition were included in these analyses. For the second analysis, we examined the proportion of transitions participants made between the objects during the restudy phase for later correct (configural memory) trials in the manipulated cue condition. Data from 18 participants were included in this analysis. There was a mean of 14 trials (SD = 6.38 trials) in the active condition and a mean of 14.17 trials (SD = 5.70 trials) in the passive condition. For the third eye analysis, we examined the proportion of transitions participants made between objects during the restudy phase as a function of subsequent configural memory for manipulated cue trials. Data from 16 participants were included in this analysis. There was a mean of 13.06 (SD = 6.12) active later-correct trials, 13.13 (SD = 3.96) active later-incorrect trials, 13.75 (SD = 5.90) passive later-correct trials, and 15.69 (SD = 3.68) passive later-incorrect trials.
We measured fMRI BOLD signal during the study and test phases using a Siemens (Erlangen, Germany) 3-T PRISMA scanner with a 64-channel head coil. BOLD fMRI was conducted using whole-brain EPI oriented with the AC–PC axis (repetition time = 2000 msec, echo time = 20 msec, field of view = 210 mm, voxel size = 1.7 mm isotropic, 70 slices, 0-mm interslice gap), with Multiband factor 2 acquisition. Structural MRI was obtained after the task to provide anatomical localization (magnetization prepared rapid gradient echo T1-weighted scans, voxel size = 1 mm isotropic, field of view = 256, 176 sagittal slices). Responses were made with the right hand using an MRI-compatible mouse. fMRI data were analyzed using AFNI (Cox, 1996). Preprocessing steps included motion correction, correction of multiband and slice-timing discrepancies, coregistration of structural and functional images, transformation to stereotactic space (Rex, Ma, & Toga, 2003), removal of linear signal drift, spatial smoothing of functional data with a 4-mm FWHM Gaussian kernel, and conversion of raw BOLD signal values to time series proportion-change values.
We conducted a univariate analysis based on conditions separated by manipulation condition (active vs. passive), later test cue (manipulated vs. nonmanipulated cue), and phase (initial study, manipulation, and restudy). Conditions were further separated based on subsequent configural memory accuracy (correct vs. incorrect), as this behavioral measure resulted in meaningful active versus passive differences (see Results). Parameter estimates corresponding to these conditions were obtained for each participant using a deconvolution approach within a general linear model (as in standard event-related fMRI designs). For the manipulation phase, trials were modeled using a duration-modulated regressor by convolving a boxcar function corresponding to the length of the trial periods with a canonical hemodynamic response function. For the initial study and restudy phases, trials were modeled using a regressor generated by convolving a boxcar function corresponding to 4-sec trial periods with a canonical hemodynamic response function. Nuisance variables for each model included T0 and T1* components of the MR signal and six-parameter movement estimates. For group-level random effects analyses incorporating two-tailed t tests, the voxel-wise threshold was set to p < .005, and a spatial extent threshold was determined using Monte Carlo simulation (Forman et al., 1995) and mixed effects multilevel analysis (Chen, Saad, Nath, Beauchamp, & Cox, 2012). The spatial extent threshold for whole-brain analyses was identified as 119 contiguous suprathreshold voxels to obtain a combined corrected threshold of p < .05 (identified for the most conservative level for any contrast and applied to all contrasts). For targeted analysis of medial-temporal lobe activity, we used a small-volume corrected extent threshold of 30 contiguous voxels, which was below the cluster size of hippocampal activity detected in our whole-brain analysis (see Results). Nonbrain voxels, including ventricles, as well as voxels that were not sampled from the brain in every participant were masked from all analyses.
For the eye movement fMRI analysis, we used the proportion of other-object transitions during restudy as a parametric regressor to identify activity associated with this eye movement behavior. Each phase (initial study, manipulation, restudy) and condition (active, passive) were modeled separately. For the connectivity analysis, we used the beta-series correlation method (Rissman, Gazzaley, & D'Esposito, 2004). Individual trials were modeled within the context of a general linear model. Parameter estimates for individual trials were sorted based on phase (initial study, manipulation, restudy) and condition (active, passive). This analysis enabled us to examine relationships between brain regions across different phases of the task. For the manipulation phase, we created a time series of beta values for the hippocampal cluster identified in the active versus passive manipulation. For the restudy phase, we created a time series of beta values for the three cortical regions that were associated with other-test transition eye movements (see Results), collapsed across right dorsolateral pFC (R-DLPFC), right inferior parietal lobe (R-IPL), and right lateral occipital cortex (R-LOC). We then correlated the hippocampal manipulation phase beta time series with the cortical restudy phase beta time series for the active and passive conditions separately for each participant. Each correlation value was then Fisher z transformed, and statistics were performed on these normalized correlation values.
To summarize the fMRI group-level analysis strategy, the following steps were performed. First, we examined activity related to subsequent configural memory accuracy for the active versus passive manipulation conditions. For this analysis, we assessed activity during manipulation and restudy, comparing active later-correct configural memory trials to passive later-correct configural memory trials. This yielded several ROIs from the manipulation phase contrast and the restudy contrast, which were further interrogated. Next, we aimed to identify activity during the restudy phase that was significantly correlated with exploratory viewing behavior. For this analysis, we used other-test transitions as a parametric modulator of brain activity during restudy for the active and passive conditions and focused on activity that was significantly modulated by eye movements using ROIs obtained from the univariate restudy contrast. Finally, we conducted a beta-series connectivity analysis using ROIs derived from the aforementioned manipulation and restudy analyses. Specifically, we correlated hippocampal activity during manipulation (predictive of subsequent memory) with frontoparietal regions during restudy (correlated with exploratory viewing behavior) to identify correlation between activity from distinct regions across phases.
Active Retrieval Effects on Subsequent Memory Accuracy
Each episode contained three objects and three grid locations (Figure 2A); therefore, active retrieval could influence memory of different types of information from the episode, including spatial information and object identity information. First, active retrieval could promote memory of the associated object identities (e.g., the cake is associated with the elephant and the ball; Figure 2B). To test this, we examined memory for associated object identities by measuring performance on the recognition test (Figure 2B). We subjected the proportion of trials in which the correct object was selected to repeated-measures ANOVA with Manipulation (active, passive) and Cue type (manipulated, other) as factors. There were no significant effects (interaction of Manipulation condition and Cue type: F(1, 20) = 0.03, p = .861; main effect of Condition: F(1, 20) = 0.66, p = .43; main effect of Cue type: F(1, 20) = 0.07, p = .80), indicating that active retrieval of one object location during the manipulation phase did not influence later memory for the associated identities of the other objects.
Second, active retrieval could promote memory of the precise object–location pairings (e.g., elephant is in the bottom left corner; ball is in the bottom right corner; Figure 2C). To test this, we examined memory for precise object–location pairings by calculating the proportion of trials on which the tested object was placed in its associated location on the grid (bull's-eye hit; Figure 2C). There was a nonsignificant trend for active retrieval to promote memory for the precise object–location associations (interaction of Manipulation condition and Cue type: F(1, 20) = 2.12, p = .161; main effect of condition: F(1, 20) = 0.70, p = .41; main effect of Cue type: F(1, 20) = 0.16, p = .70). Therefore, although there was a trend, active retrieval did not significantly promote memory of the precise object–location pairings from the episode.
Finally, active retrieval could promote memory of the overall spatial configuration (e.g., the other objects are located in the bottom two corners; Figure 2D). We therefore tested the influence of active retrieval on memory for the spatial configuration of each episode (Figure 2D). We reasoned that active retrieval would promote memory of the other object locations associated with the actively retrieved object, because the manipulation involved recalling a specific location from the episode, which could have caused participants to focus on the spatial arrangement of the objects. We defined configural recall accuracy as proportion of trials in which the tested object was placed in its precise location or the nontested object's location from that trial (Figure 2E; see Methods). As predicted, configural memory accuracy varied significantly by active/passive manipulation condition and manipulated/other retrieval cue type (interaction: F(1, 20) = 5.99, p = .024). When manipulated objects from the active condition served as retrieval cues, spatial recall of the overall configuration was superior compared with when manipulated objects from the passive condition were cues (t(20) = 2.54, p = .019) or when other objects from the active condition were cues (t(20) = 3.22, p = .004). Memory did not differ for manipulated/other cue types in the passive condition (t(20) = 0.80, p = .435) and for active/passive conditions when given other cues (t(20) = 0.87, p = .393). Active retrieval thus enhanced binding of the retrieved object with other object locations, such that it later served as a better retrieval cue for the entire spatial configuration (i.e., better recall of other object locations that were not themselves actively retrieved). Notably, this pattern could not have been due merely to better memory for any of the objects encoded within an active trial, as accuracy did not improve when the nonmanipulated objects from active trials were used as cues.
These results indicate that the active manipulation employed here benefited memory for the overall spatial layout rather than specifically promoting memory for the precise object–location pairings. Active retrieval of one object influenced memory for the entire spatial configuration and the interrelationships between the other objects and the manipulated object, while having minimal effects on associative recognition and precise object–location pairings of the other objects.
Hippocampal Activity during Active Retrieval Predicts Disproportionate Binding
Given the effects of active retrieval on memory for the episodic spatial configuration, we used this behavioral measure to identify specific activity associated with active retrieval versus passive reexposure. To identify neural correlates corresponding to the effect of active retrieval on memory for the overall spatial configuration, we compared fMRI activity during the manipulation phase for the active versus passive conditions for trials that were later correct (spatial configuration accuracy). Three active > passive activity clusters were identified, including hippocampus (Figure 2F), left inferior frontal gyrus (IFG), and left insula (Table 1).
|Region .||Hemisphere .||Volume (mm3) .||x .||y .||z .||Brodmann Area .|
|Active > Passive during the manipulation phase|
|Active > Passive during the restudy phase|
|Superior parietal lobe||Left||1080||−15||−71||55||7|
|Lateral occipital gyrus||Right||705||36||−87||10||19|
|Medial frontal gyrus||Bilateral||584||3||31||41||6|
|Medial frontal gyrus||Bilateral||564||4||41||29||9|
|Region .||Hemisphere .||Volume (mm3) .||x .||y .||z .||Brodmann Area .|
|Active > Passive during the manipulation phase|
|Active > Passive during the restudy phase|
|Superior parietal lobe||Left||1080||−15||−71||55||7|
|Lateral occipital gyrus||Right||705||36||−87||10||19|
|Medial frontal gyrus||Bilateral||584||3||31||41||6|
|Medial frontal gyrus||Bilateral||564||4||41||29||9|
Montreal Neurological Institute coordinates are the centroids of the clusters. Activity identified by targeted analysis of medial temporal lobe regions denoted by a footnote symbol (a).
We further interrogated the hippocampal cluster to identify activity associated with subsequent memory in the active condition. We hypothesized that hippocampus would be sensitive to binding in the active manipulated cue condition, as activity during retrieval may predict strong binding between the retrieved object and the other objects, but not necessarily binding between the other objects. Therefore, we conducted targeted analyses of subsequent memory effects as a function of later cue condition in the active condition. Hippocampal activity was greater for manipulated cue trials that were later correct (M = 0.05, SE = 0.02) compared with later-incorrect trials (M = 0.00, SE = 0.02; t(19) = 2.48, p = .023). On the other hand, in the active nonmanipulated cue condition, hippocampal activity during the active manipulation did not differentiate between later-correct (M = 0.007, SE = 0.04) and later-incorrect (M = 0.017, SE = 0.02) trials (t(19) = 0.19, p = .85). Hippocampal activity did not differ for later-correct manipulated versus nonmanipulated trials (t(19) = 0.19, p = .85), indicating that hippocampus was not selective to binding in the active manipulated cue condition only. Instead, these results show that hippocampus is associated with successful retrieval during active retrieval versus passive reexposure. Furthermore, this pattern of hippocampal activity enhancement for the active condition predictive of memory accuracy in the later-manipulated cue condition provides some supporting evidence that right hippocampus supported retrieval of object–location associations in a manner that contributed to binding with other objects during the upcoming restudy phase.
We also interrogated the two prefrontal clusters to identify any subsequent memory effects for the active condition. Unlike the hippocampal cluster, no subsequent memory effects arose in the left IFG or left insula (ts < 1.63, ps > .12). These results suggest that these two prefrontal regions did not track binding in the active condition, regardless of later cue condition. On the other hand, hippocampus was predictive of subsequent memory in the active manipulated cue condition only.
Exploration-related Eye Movements
We hypothesized that memory retrieval would trigger the implementation of eye movement strategies during learning of other information (Figure 3). Retrieval of select information could prompt a number of viewing behaviors that either facilitate or impair memory of the retrieved information and the other information. First, it is possible that retrieval simply promotes memory of retrieved information (i.e., the testing effect; Karpicke & Roediger, 2008; Runquist, 1983; Landauer & Bjork, 1978). In this scenario, we would expect that viewing would be primarily directed to the retrieved object, whereas the other objects would be largely ignored (Figure 3A). This type of viewing behavior would be strategic if the only goal was to later remember the retrieved information, but it would be detrimental to memory of the other information and likely would not support memory for the entire spatial configuration associated with the retrieved object. A second possibility is that the retrieved object could be used as a visual anchor to learn and associate the other objects. In this scenario, participants would implement a viewing strategy that involves making iterative fixations between the retrieved objects and the other objects (Figure 3B). This type of strategy could be expected to boost binding between the retrieved object and the other objects. A third type of viewing strategy would involve diverting attention away from the retrieved object and making iterative eye movements between the other objects (eye movements that exclude the retrieved object; Figure 3C). Presumably, this type of strategy would promote memory of the other objects and the association between those objects but not between the other objects and the retrieved object. However, we hypothesize that retrieval causes information to enter a temporary active state, so even if visual attention is directed elsewhere, it remains active for a temporary period. Therefore, this third type of viewing behavior could be effective in promoting memory for the retrieved object and the other objects if the retrieved object is maintained in an active state during viewing of the other objects.
We evaluated these three possible eye movement behaviors after active retrieval versus passive reexposure that could indicate strategic deployment of viewing after the manipulation. To evaluate the first hypothesis (Figure 3A; retrieved information dominates attention), we examined whether the overall proportion of viewing the manipulated object differed after the active versus passive condition. Counter to the hypothesis that active retrieval causes participants to focus attention solely on the retrieved object, the overall proportion of time spent viewing the manipulated object during the restudy phase tended to be less after the active (M = 0.30, SE = 0.01) versus passive (M = 0.33, SE = 0.01) condition (t(20) = 1.93, p = .067).
We next asked whether viewing of the manipulated object varied over the course of the restudy phase. We therefore subdivided the proportion of time viewing manipulated objects into a time course of eight 500-msec intervals (Figure 3D; cf. Hannula & Ranganath, 2009). Manipulated object viewing differed significantly for the active compared with passive conditions (main effect of condition: F(3.47, 69.34) = 4.04, p = .008GG). Post hoc testing indicated significantly less viewing in the active relative to the passive condition from 1501 to 3000 msec (1501–2000 msec: t(20) = 3.05, p = .006; 2001–2500 msec: t(20) = 2.82, p = .011; 2501–3000 msec: t(20) = 2.76, p = .012; all other intervals: ps > .10). Taken together with the overall proportion of viewing time analysis, these results indicate that participants did not direct their focus of attention to the retrieved object but instead diverted their gaze to the other objects on the screen after the active versus passive manipulation.
We next tested Hypotheses 2 (Figure 3B; retrieved information serves as a visual anchor) and 3 (Figure 3C; retrieved information is ignored because it is in active memory) by examining object-to-object transitions, which have been linked to memory-guided exploration (Redish, 2016; Wang et al., 2015; Voss, Warren, et al., 2011; Tolman, 1932). For this analysis, we focused our attention on manipulated cue trials that were later correct (configural memory accuracy). We examined the proportion of transitions participants made between the manipulated object and the later-tested object. More transitions between the manipulated object (cue) and the later tested object after active retrieval would provide support for the visual anchor hypothesis, as participants would use the retrieved object as the anchor to look between the other objects. To test the active memory hypothesis, we examined the proportion of transitions between the two other, nonmanipulated objects. More other-test transitions in the active condition would reflect strategic allocation of learning resources to the nonretrieved objects, while the retrieved object remained active in memory. To directly compare these competing hypothesis, we subjected the proportion of transitions to a repeated-measures ANOVA with Transition type (other test/manipulated test) and Manipulation condition (active/passive) as factors (Figure 3E). A significant interaction of Transition type and Manipulation condition (F(1, 17) = 5.35, p = .034) indicated that participants made more other-test transitions relative to manipulated-test transitions in the active condition (t(17) = 2.47, p = .024), whereas Transition type did not vary in the passive condition (t(17) = 1.21, p = .241). Furthermore, participants made significantly more other-test transitions in the active relative to the passive condition (t(17) = 4.37, p = .0004; Figure 4E). These results provide evidence supporting the active memory hypothesis, given that participants diverted their attention away from the retrieved object, although it later served as a strong cue to recall the other objects' locations. That is, although participants made more other-test transitions in the active condition, the other (nonmanipulated) object did not serve as a strong cue to later recall the tested object. These differential viewing patterns across conditions support the interpretation that active retrieval strategically enhanced viewing of nonretrieved objects to promote learning. Coupled with the memory benefit achieved when manipulated objects were test cues in the active condition only, these results suggest that other-test transitions served to increase integration of the retrieved object with the other objects that were viewed.
This interpretation was further supported by an analysis of the relationship between iterative viewing patterns and subsequent memory (see Methods). For this analysis, we examined the proportion of other-test transitions for later manipulated cue trials (Figure 3F). The proportion of other-test transitions varied significantly by condition (active/passive) and subsequent memory accuracy (correct/incorrect; F(1, 15) = 5.29, p = .036). Whereas other-test transitions were detrimental to memory in the passive condition, in that they were more prevalent for later-incorrect compared with correct trials (t(15) = 2.43, p < .026), the proportion of other-test transitions did not differ between correct and incorrect trials in the active condition (t(15) = 0.47, p = .645). On the other hand, manipulated-test transitions did not vary significantly by condition and memory accuracy (F(1, 15) = 0.28, p = .603). Thus, in the passive condition, other-test viewing behavior was detrimental to later memory for the retrieved item because viewing was directed at the other items at the expense of the retrieved item. This was not the case in the active condition, because the retrieved item was in active memory during other-test viewing and was therefore protected against the detrimental effects of reduced viewing on later memory. Active retrieval thus protected the actively retrieved information from the damaging effects of reduced viewing (i.e., other-test transitions) selectively in the active condition.
These results support the interpretation that other-test transitions were only strategic after active retrieval and not passive reexposure, because attentional resources needed to be distributed across all objects in the passive condition (not only allocated toward nonmanipulated objects). In contrast, because the manipulated object was recently retrieved in the active condition, resources could be efficiently allocated to the other objects, although it remained available for binding as participants iteratively viewed the other objects (Figure 3C).
Frontoparietal Network Activity during Strategic Exploration
To identify corresponding fMRI activity correlates, we tested for activity that varied parametrically from trial to trial with the proportion of other-test viewing transitions (Figure 4A), restricted to regions broadly implicated in active memory effects during restudy (i.e., active/passive activity contrast ROIs for later-correct trials; see Methods; Table 1). This parametric analysis for the active condition identified regions in the frontoparietal network, including R-DLPFC: t(19) = 2.38, p = .028 (M = 0.104, SE = 0.044); R-IPL: t(19) = 2.60, p = .018 (M = 0.083, SE = 0.032); and R-LOC: t(19) = 3.46, p = .003 (M = 0.149, SE = 0.043; Figure 4B). Activity did not vary with other-test viewing transitions for any activity cluster in the passive condition (R-DLPFC: t(19) = 0.260, p = .798 [M = 0.018, SE = 0.071]; R-IPL: t(19) = 0.760, p = .457 [M = 0.028, SE = 0.036]; R-LOC: t(19) = 0.917, p = .371 [M = 0.045, SE = 0.049]). Importantly, this analysis was based on the proportion of other-test eye movement transitions that were made after active versus passive manipulation, not on the number of object-to-object transitions. Therefore, this activity did not reflect the simple implementation of eye movement transitions but corresponded to a specific behavioral strategy that was only used after active retrieval (higher proportion of other-test transitions vs. other types of transitions). These results suggest that the frontoparietal network was involved in strategic other-test viewing transitions after active retrieval only.
Hippocampus Predicts Frontoparietal Network Activity Associated with Implementation of Strategic Eye Movements
We next tested whether hippocampal activity associated with active retrieval during the manipulation phase predicted frontoparietal exploration-related activity during the subsequent restudy phase. Greater hippocampal activity during active retrieval predictive of enhanced activity in the frontoparietal network during restudy would implicate hippocampus in permitting the cortical network to implement strategic eye movements. We evaluated this relationship using a novel time-lagged beta-series correlation analysis (see Methods). Relative to the passive condition, hippocampal activity during the active manipulation phase significantly correlated with activity in the frontoparietal network during the restudy phase (t(19) = 2.24, p = .038; Figure 5). Thus, hippocampal activity reflecting active retrieval significantly predicted subsequent activity reflecting strategic exploration eye movement patterns during restudy. Time-lagged coordination between the hippocampus and the frontoparietal network was associated with cortically mediated implementation of exploratory viewing strategies during restudy, which contributed to disproportionate binding between the actively retrieved information and the other episodic elements.
We designed a memory task that artificially segregated active retrieval from memory-guided exploration. In a spatial memory task, participants studied multiobject location associations and then selectively retrieved one object. We examined viewing behavior after the manipulation, when participants restudied all three objects. Active retrieval of one object promoted memory of the entire spatial configuration of the episode, only when the retrieved object was given as a cue on the final test. After active retrieval, but not after a passive control condition, participants implemented an exploration strategy that involved directing attention away from the retrieved object and making iterative transitions between the nonretrieved objects. We propose that the retrieved object was active in memory after the retrieval manipulation, enabling participants to ignore the retrieved object, while focusing attentional resources to the other objects. Our results showed that active retrieval influences not only memory for the retrieved material but also how other information in the episode is learned and incorporated into a cohesive memory representation.
Hippocampal activity was linked to successful memory retrieval, whereas frontoparietal activity was associated with strategic viewing patterns triggered by retrieval. This constitutes the first within-experiment spatiotemporal dissociation involving hippocampal contributions to memory retrieval versus frontoparietal contributions to memory-guided exploration strategies. Notably, hippocampal activity during active retrieval predicted immediately upcoming frontoparietal activity compared with the passive reexposure condition, showing that the extent of hippocampal engagement during memory retrieval predicted frontoparietal engagement during the implementation of memory-guided exploratory strategies. Thus, active retrieval shaped learning by dictating how other information in the episode was explored and therefore how it was stored in memory.
The current evidence for distinct yet interactive roles of hippocampus and frontoparietal networks in memory-guided exploration builds on previous findings of hippocampal and frontoparietal interactions for memory-based strategic exploration during learning (Wang et al., 2015; Wang & Voss, 2014; Yee et al., 2014; Voss, Gonsalves, et al., 2011; Voss, Warren, et al., 2011). For instance, during object–location learning, hippocampal–cortical activity was associated with an iterative viewing strategy that enhanced memory for specific objects and their locations (Voss, Gonsalves, et al., 2011; Voss, Warren, et al., 2011). The crucial role of hippocampal-dependent memory in guiding exploration was supported by two findings: (1) Individuals with hippocampal amnesia did not benefit from having active control of exploration (despite overall above-chance performance; Voss, Gonsalves, et al., 2011), and (2) these amnestic patients did not engage in strategic exploration behaviors (Voss, Warren, et al., 2011), indicating that exploration was memory dependent. Although these previous findings indicate that memory retrieval and strategic exploration are interactive and supported by hippocampal–cortical interactions, they were unable to isolate specific memory retrieval events and the exploration patterns that they trigger. The iterative viewing strategy described here (Figure 3C) is functionally similar to the strategy reported in Voss, Warren, et al. (2011; see also Redish, 2016), and the current findings further include clear specification of the effects of retrieval on exploration strategies and the resultant binding that occurred with retrieved elements. The current results are unique in showing that hypothesized interactions between memory and exploration can be directly observed in viewing behavior and thereby localized to relevant brain activity. Hippocampal activity indicative of memory retrieval correlated with frontoparietal correlates of subsequent exploration triggered by this retrieval. These findings pinpoint the role of hippocampal–frontoparietal interactivity in memory's influence on exploration and suggest network level mechanisms for interaction of memory with exploration and learning.
Disproportionate binding of the retrieved object to the other objects' locations occurred despite less overall viewing of the retrieved element and fewer direct transitions between the retrieved object and the other object that was later tested. We propose that this occurred because memory retrieval caused content to enter an “active memory” state (Lewis, 1979) such that it was available for binding with other content that was sampled during learning, despite being viewed less itself (Bridge & Voss, 2015). In this way, memory retrieval shaped the resultant memory representation by structuring episodic memories around the retrieved content that guided exploration behavior. This was accomplished via active visual exploration of content that was not in the active memory state (Figure 3C). Although other-test transitions that excluded the manipulated object were actually detrimental to memory in the passive condition, memory for the manipulated-test object association was protected in the active condition, because the manipulated object was in active memory during the implementation of this sampling behavior. We propose that preferential sampling of other objects allowed the retrieved object to be repeatedly bound with sampled content and therefore caused it to become central to resulting memory representations. This suggests a covert form of binding between the retrieved object and other objects, which we interpret as a signal that the retrieved object was in active memory for a temporary period (Lewis, 1979) and therefore available for binding even when not directly viewed. Consistent with this interpretation, functional connectivity analyses indicated that hippocampal activity corresponding to memory retrieval (and therefore the entrance of objects into active memory) predicted immediately upcoming activity correlates of exploration behavior in the frontoparietal network. Thus, active retrieval provided a window during which enhanced binding with the retrieved information could occur and influenced exploration strategies, thus determining what information would be bound to the active memory trace. These findings show that retrieval and exploration interact to form and structure episodic memories, associated with interactive hippocampal cortical regions implicated in retrieval-induced binding and strategic exploration.
Testing memory for previously learned material enhances later memory compared with passive reexposure to the same information (Karpicke & Roediger, 2008; Runquist, 1983; Hogan & Kintsch, 1971). Unlike previous experiments on fMRI correlates of such retrieval influences on memory (Nelson, Arnold, Gilmore, & McDermott, 2013; Wing, Marsh, & Cabeza, 2013), the current study assessed influences of retrieval on memory for interrelationships among elements of an episode rather than for the specific retrieved material itself. Eye tracking provided measurement of attentional focus with high precision across space and time. Although participants viewed other, nonretrieved objects more so than the retrieved object after retrieval, the retrieved object was nonetheless disproportionately bound with the viewed objects. Our interpretation of this effect is an indication that retrieved content was temporarily in an active memory state (Lewis, 1979) and therefore available for binding despite attention being directed elsewhere, which is consistent with previous studies showing maintenance of attended information across distractor periods. Multivariate pattern analysis of fMRI data (Larocque, Lewis-Peacock, & Postle, 2014; Lewis-Peacock, Drysdale, Oberauer, & Postle, 2012) has shown that information can be maintained over a delay period even when a distractor shifts attentional focus away to task-irrelevant information. These findings provide converging evidence that information can be maintained in an active state even when the focus of visuospatial attention is elsewhere (Lewis-Peacock et al., 2012). Although these studies were unable to identify specific neural correlates of maintaining information in an active state outside the focus of attention (Larocque et al., 2014; LaRocque, Lewis-Peacock, Drysdale, Oberauer, & Postle, 2013; Lewis-Peacock et al., 2012), this information was available for rapid access compared with inactive information in long-term memory (Larocque et al., 2014), therefore indirectly yielding evidence for active maintenance. Here, we provide evidence that hippocampus is involved in the entrance of retrieved information into the active memory state, allowing binding with other information when available (as in our study), and perhaps maintenance across delays when necessary (as in previous studies).
A caveat of these retrieval effects is that active retrieval supported memory for the overall spatial configuration but not associative recognition or precise object–location memory. Although there was a nonsignificant trend for active retrieval to promote memory for the precise object–location associations, the only robust effects of active retrieval were on memory for the overall spatial configuration of the objects. The analysis of eye movements during the restudy phase after active retrieval indicated that participants alternated their gaze between the other objects on the grid, presumably while the retrieved object was active in memory. It is possible that this eye movement pattern supported memory for the overall configuration of the objects on the grid by increasing memory for the fixation pattern that was made during viewing itself, without enhancing memory for the precise object–location associations as robustly. Therefore, the eye movement behavior generated during restudy may have contributed to the type of memory representation that was formed.
Active retrieval may have been more engaging than passive reexposure, although both conditions involved using the mouse to select an object and move it to its associated location on the grid. Furthermore, trial order was randomized, and participants did not know the condition for a specific trial until after the initial study portion. Nonetheless, it is possible that greater attention was a factor in the effects of active retrieval, given the established association between attention and learning (Chun & Turk-Browne, 2007). However, the pattern of findings would be very hard to reconcile with a simple attention account. First, memory was not overall better in the active condition, but rather, memory for nonmanipulated objects benefited after active retrieval only when an active manipulated object was used as a test cue (not when an active nonmanipulated object was a test cue). This selective effect of active retrieval on the association between the retrieved object and other objects does not fit with the notion that attention was nonspecifically enhanced in the active condition, as presumably memory for all objects would benefit equally because of heightened attention. Furthermore, the time spent viewing any of the three objects did not differ during the restudy period for the active compared with passive conditions (see Eye-tracking Methods section), which is also inconsistent with active retrieval causing heightened visuospatial attention. Finally, we found that the specific viewing strategy employed after active versus passive retrieval was a critical component of the beneficial effects on memory. This specific pattern of viewing linked to correspondingly specific effects on memory would not be expected under a heightened attention account. Thus, although many factors could potentially vary between the active and passive conditions, including attention, our results suggest that the most salient factors contributing to later memory are retrieval and subsequent viewing strategies triggered by retrieval, not other nonspecific variables.
This experiment adds to theoretical accounts of the dynamic influences memory retrieval has on new learning. Findings support theories of hippocampal–cortical connectivity involvement in the generation of exploratory behaviors based on memory content (Redish, 2016; Wang et al., 2015). These theories propose that cortical regions generate information-gathering behaviors based on the content of ongoing hippocampal-dependent memory retrieval, thus yielding behavior that is strategic in the sense that it is tuned to relevant information based on what has already been learned. Indeed, the current findings demonstrate hippocampal interaction with frontoparietal cortex in the service of strategic, exploratory viewing that forms cohesive memory representations centered on retrieved memory content. Although the current approach measured retrieval influences on exploratory behavior and its ramifications for the structure of resulting memories (and corresponding fMRI correlates) with greater specificity than has been achieved in previous studies, future work using methods with greater anatomical and temporal precision than fMRI, such as intracranial electrophysiological recordings from the hippocampus and cortex, will be needed to identify the fine-grained dynamics of hippocampal–cortical interactions supporting retrieval influences on exploration.
Financial support was provided by grants F32NS087885 and R00NS069788 from the National Institute of Neurological Disorders and Stroke and grants R21MH108863 and R01MH062500 from the National Institute of Mental Health. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We thank Elise Gagnon and Bianca Marin for assistance with data collection and Peter Winter for helpful discussion. Neuroimaging was performed at the Center for Translational Imaging, supported by the Northwestern University Department of Radiology. This research was supported in part through the computational resources and staff contributions provided for the Quest high-performance computing facility at Northwestern University, which is jointly supported by the Office of the Provost, the Office for Research, and Northwestern University Information Technology.
Reprint requests should be sent to Donna J. Bridge, Northwestern University Feinberg School of Medicine, Laboratory for Human Neuroscience, Ward 18-134, 303 E. Chicago Ave., Chicago, IL 60611, or via e-mail: firstname.lastname@example.org.