Abstract

According to the principle of reactivation, memory retrieval evokes patterns of brain activity that resemble those instantiated when an event was first experienced. Intuitively, one would expect neural reactivation to contribute to recollection (i.e., the vivid impression of reliving past events), but evidence of a direct relationship between the subjective quality of recollection and multiregional reactivation of item-specific neural patterns is lacking. The current study assessed this relationship using fMRI to measure brain activity as participants viewed and mentally replayed a set of short videos. We used multivoxel pattern analysis to train a classifier to identify individual videos based on brain activity evoked during perception and tested how accurately the classifier could distinguish among videos during mental replay. Classification accuracy correlated positively with memory vividness, indicating that the specificity of multivariate brain patterns observed during memory retrieval was related to the subjective quality of a memory. In addition, we identified a set of brain regions whose univariate activity during retrieval predicted both memory vividness and the strength of the classifier's prediction irrespective of the particular video that was retrieved. Our results establish distributed patterns of neural reactivation as a valid and objective marker of the quality of recollection.

INTRODUCTION

Recollection allows people to mentally travel back in time to relive past events. The recollective experience is personal and subjective, and thus historically, it has been difficult to capture with objective measures. However, the recent application of multivoxel pattern analysis (MVPA; Haxby, 2012; Mahmoudi, Takerkart, Regragui, Boussaoud, & Brovelli, 2012; Tong & Pratte, 2012; Haynes & Rees, 2006; Norman, Polyn, Detre, & Haxby, 2006) to functional brain imaging data provides a way of examining recollection from an objective standpoint. MVPA can quantify cortical reinstatement or reactivation (Rissman & Wagner, 2012; Danker & Anderson, 2010; Rugg, Johnson, Park, & Uncapher, 2008), the phenomenon by which stimulus-specific patterns of brain activation elicited at perception are reactivated during subsequent memory retrieval. Although reactivation by itself does not guarantee the phenomenological experience of recollection (Johnson, McDuff, Rugg, & Norman, 2009), we propose that reactivation is a prerequisite for a faithful and vivid recollective experience because it reflects the specificity with which stimuli are represented in memory. With the current study, we tested this proposition by examining the correspondence between distributed patterns of stimulus-specific neural reactivation and the subjective quality of the recollective experience.

A few studies have reported a link between MVPA measures of cortical reinstatement and the subjective experience of recollection (Gordon, Rissman, Kiani, & Wagner, 2013; Ritchey, Wing, Labar, & Cabeza, 2013; Staresina, Henson, Kriegeskorte, & Alink, 2012; Kuhl, Rissman, Chun, & Wagner, 2011; Johnson et al., 2009; McDuff, Frankel, & Norman, 2009). However, most of these studies have assessed recollection as an all-or-none phenomenon that is either present or absent (e.g., with a remember/know paradigm; Tulving, 1985). For example, Johnson et al. (2009) reported significantly greater reactivation when an event is “remembered” (recollected) than when it is “known” (recognized with a sense of familiarity but not recollected). Although these results are compelling, very few studies have assessed whether reactivation also reflects the graded nature of conscious memory retrieval. Although the presence and absence of recollection at retrieval is thought to reflect the engagement of qualitatively different processes (Yonelinas, Aly, Wang, & Koen, 2010; Yonelinas, 2002), recollection also results in a complex range of experiences that vary in vividness and level of detail (e.g., Rubin, Schrauf, & Greenberg, 2003). So far, only a few reactivation studies have sampled this range of “above-threshold” recollective experience. For example, Leiker and Johnson (2014) have shown that task-specific reactivation improves when more information is recollected. Johnson, Kuhl, Mitchell, Ankudowich, and Durbin (in press) have shown that aging influences patterns of correlations between classification accuracy and vividness of recall across brain ROIs. Wing, Ritchey, and Cabeza (2015) have also shown that stimulus-specific reactivation increases linearly as a function of the number of details recalled for consciously retrieved images. These findings suggest that recollection and reactivation are different facets (one subjective, one objective) of the same underlying brain processes, but more evidence is needed.

With the current study, our aim was to further explore the relationship between distributed patterns of stimulus-specific reactivation and the graded nature of conscious memory retrieval. We used complex audiovisual stimuli (short videos) to elicit distributed patterns of stimulus-specific activation and applied MVPA to capture how these patterns were reactivated across the entire brain during a recall task. In addition to being visually rich, the videos contained dynamic features such as motion and a simple “storyline” as well as complex auditory features including speech, music, and natural sounds that elicited activation throughout the cerebral mantle. Each video was encoded and mentally replayed multiple times while participants underwent fMRI. Repetition ensured that memory was strong and detailed to sample a range of “above-threshold” recollective experience, which participants qualified using a graded vividness scale. In addition, repeated retrieval made it possible to assess within-item fluctuations in vividness and reactivation, that is, each video served as its own baseline so that we could assess how well a memory for the same stimulus was reconstructed from trial to trial. In this manner, we controlled for the influence of interitem factors on vividness and reactivation (e.g., differences in memory strength, level of interest in each item).

We hypothesized that a person's subjective evaluation of the quality of recollection is a kind of “readout” of the extent to which a distributed neural trace formed during perception is reactivated at recall. Therefore, we expected subjective vividness ratings to predict trial-specific MVPA scores that indicated how well stimulus-specific activity patterns were reactivated during mental replay. We also predicted that brain regions whose activity correlated with (subjective) vividness should also correlate with the (objective) specificity of neural reactivation during recall, indicating underlying mechanisms common to recollection and reactivation. To test this hypothesis, we first identified brain regions whose activity correlated with vividness ratings across videos. Then, we identified brain regions whose activity correlated with reactivation specificity, using an approach that we call the local–global analysis (LGA). With LGA, we correlated trial-specific whole-brain (global) MVPA reactivation scores with voxelwise (local) levels of univariate activity during mental replay (see also Xue et al., 2013; Kuhl et al., 2011; Li, Mayhew, & Kourtzi, 2009, for approaches that combine “activation-based” and “information-based” analyses). We identified voxels whose level of activity correlated with reactivation across videos, which we compared with voxels identified by the vividness analysis using a conjunction mask.

Johnson and colleagues (Johnson, Suzuki, & Rugg, 2013; Johnson & Rugg, 2007) distinguish between two different categories of brain structures involved in recollection: (1) content-sensitive structures whose activity distinguishes among stimulus items (e.g., areas from the ventral visual stream that support stimulus representation) and (2) structures involved in core recollective processes that are engaged regardless of the content retrieved (e.g., the inferior lateral parietal cortex; Johnson et al., 2013; Quamme, Weiss, & Norman, 2010; Hutchinson, Uncapher, & Wagner, 2009; Vilberg & Rugg, 2007; Wagner, Shannon, Kahn, & Buckner, 2005; but see Kuhl & Chun, 2014; Vilberg & Rugg, 2008). MVPA has been used successfully to identify brain structures that fall under the first category, that is, structures whose activity at retrieval is content sensitive. On the other hand, only a few studies have addressed how content-general memory processes promote the distributed reactivation of stimulus-specific patterns during memory retrieval, and most studies have focused on a narrow subset of brain regions. For example, hippocampal activation has been shown to promote the cortical reinstatement of scene-specific (Ritchey et al., 2013) and categorical source information (such as faces vs. scenes; Gordon et al., 2013) and of low-level visual gratings (Bosch, Jehee, Fernandez, & Doeller, 2014).

Here, we used LGA to assess how activity in each brain voxel correlated with patterns of reactivation distributed across the entire brain. We assumed that our whole-brain MVPA reactivation measure was largely driven by activity from content-sensitive structures. We hypothesized, however, that regions whose univariate activity correlated with vividness and whole-brain reactivation metrics across all videos should reflect core recollective processes engaged irrespective of stimulus content (e.g., content-general attentional, retrieval, or control processes). In summary, we predicted that specific neural representation and the vivid impression of reliving past events should be fundamentally related and that both phenomena should rely on a common set of brain regions that play a content-general role in recollection.

METHODS

Participants

Nineteen adult participants (5 men and 14 women, 20–33 years old, 14–18 years of education) were recruited, tested, and paid for their participation according to a protocol approved by the Rotman Research Institute's research ethics board. All participants were right-handed and either native or fluent English speakers, with normal hearing, normal or corrected-to-normal vision, no history of neurological or psychiatric disease, and no other contraindications for MRI. Data from 14 of these participants were reported as part of a different study on memory and aging by St-Laurent, Abdi, Bondad, & Buchsbaum, (2014).

fMRI Task

Video Stimuli

Fifteen short video clips were gathered from online sites (Vimeo.com and YouTube.com), edited, and exported to an .avi format with iMovie. Four clips were used for practice, and 11 were used for the in-scan task (Figure 1). Each video was associated with a short descriptive title shown in conjunction with the video that served as a retrieval cue during mental replay trials.

Figure 1. 

(Top) In-scan cued recall paradigm. Teal-colored titles were always followed by the matching video, whereas red titles were followed by a gray rectangle. Participants mentally replayed the video corresponding to the title while the rectangle was displayed on the screen and then rated the quality of their replay from 1 to 4. (Bottom) Title, screen-shot, and sound track description for the set of 11 videos viewed and replayed by the participants (figure from St-Laurent et al., 2014).

Figure 1. 

(Top) In-scan cued recall paradigm. Teal-colored titles were always followed by the matching video, whereas red titles were followed by a gray rectangle. Participants mentally replayed the video corresponding to the title while the rectangle was displayed on the screen and then rated the quality of their replay from 1 to 4. (Bottom) Title, screen-shot, and sound track description for the set of 11 videos viewed and replayed by the participants (figure from St-Laurent et al., 2014).

Testing Procedure

Participants performed a cued recall task while undergoing fMRI scanning. They viewed and recalled, or mentally replayed, the same set of 11 short videos over seven functional runs of 8 min each. Perception and mental replay trials were intermixed throughout the runs according to a pseudorandom order that differed between runs and participants. For a particular video, perception and replay trials always alternated, with the restriction that they were separated by at least two (and maximum eight) intervening trials. Each of the videos were 3.4-sec long (plus 0.2 sec of buffer time to load), and they were shown and recalled 21 times each (three times per run). All trials were cued by a short title that matched a video (e.g., “Race Car,” “Skateboarding Dog”). During perception trials, a title was shown in teal letters (0.9 sec) in the center of the screen, followed by the matching video (3.6 sec) and by an ISI of 0.25–1 sec. During mental replay trials, a title was shown in red letters (0.9 sec), followed by a gray rectangle that covered the same portion of the screen as the videos shown on perception trials (4.85 sec). While the gray rectangle was on the screen, participants were instructed to mentally replay the video from memory as vividly and with as many visual and auditory details as possible. Participants were then given 2.75 sec to rate the vividness of their replay on a 1–4 scale (1 = not vivid at all, between 2 and 3 = average, and 4 = extremely vivid), followed by an ISI of 0–0.75 sec. Participants practiced the task with a separate set of four videos before scanning and during an anatomical scan acquired immediately before the first functional run. The experimental stimulus set was shown for the first time during the first functional run. After the scan, participants' memory for the content of the 11 videos was tested with a recall and a recognition task (see St-Laurent et al., 2014, for procedure and results).

MRI Setup and Data Acquisition

Stimuli and responses were presented and recorded using E-Prime 2.0 (Psychology Software Tools, Pittsburgh, PA). Visual stimuli were projected onto a screen behind the scanner made visible to the participant through a mirror mounted on the head coil. Audio stimuli were delivered from the PC running the experimental task through electrodynamic headphones using the MR-Confon MRI-compatible audio system. Participants were scanned with a 3.0-T Siemens (Erlangen, Germany) MAGNETOM Trio MRI scanner using a 12-channel head coil system. High-resolution gradient-echo multislice T1-weighted scans (160 slices of 1-mm thickness, 19.2 × 25.6 cm field of view) coplanar with the EPI scans as well as whole-brain magnetization prepared rapid gradient-echo (MP-RAGE) 3-D T1-weighted scans were first acquired for anatomical localization, followed by T2*-weighted EPIs sensitive to BOLD contrast. Images were acquired using a two-shot gradient-echo EPI sequence (22.5 × 22.5 cm field of view with a 96 × 96 matrix size, resulting in an in-plane resolution of 2.35 × 2.35 mm for each of 26 3.5-mm axial slices with a 0.5-mm interslice gap; repetition time = 1.5 sec; echo time = 27 msec; flip angle = 62°).

fMRI Data Analysis

All statistical analyses were first conducted on smoothed and realigned functional images in native EPI space. The MP-RAGE anatomical scan was normalized to the Montreal Neurological Institute (MNI) space using nonlinear symmetric normalization implemented in ANTS (Avants, Epstein, Grossman, & Gee, 2008). An equivalent transformation was then applied to maps of statistical results derived from functional images using ANTS to normalize these maps for group analyses.

Univariate fMRI Analysis

Functional images were converted into NIFTI-1 format, motion-corrected, and realigned to the first image of the first run with AFNI's (Cox, 1996) 3dvolreg program and smoothed with a 4-mm FWHM Gaussian kernel. Single-subject multiple regression modeling was carried out separately for each functional run using the AFNI program 3dDeconvolve. Within each run, the two conditions (video perception and mental replay) were modeled separately for each trial by convolving a hemodynamic response function (SPM canonical function as implemented in AFNI) with the onset and the duration of each experimental event. For each encoding and each mental replay trial, this procedure generated beta coefficients that were used for training and testing an MVPA classifier on individual perception and mental replay events (Rissman, Gazzaley, & D'Esposito, 2004). Brain activity evoked during the vividness judgment after mental replay was also modeled but not analyzed. A set of five nuisance regressors (a constant term plus linear, quadratic, and higher order polynomial terms) was also included for each scanning run to model low-frequency noise in the time series data.

Pattern Classification

We trained two pattern classifiers to determine which video from the stimulus set was recalled during mental replay trials. The recall-trained classifier was trained via leave-one-run-out cross-validation on mental replay trials. The perception-trained classifier was trained on all perception trials and then used to “cross-decode” (Kriegeskorte, 2011) recall trials. The type of classification we used was shrinkage discriminant analysis (SDA version 1.3.3; Strimmer Lab, Imperial College London; strimmerlab.org/software/sda), a form of regularized linear discriminant analysis that can be applied to high-dimensional data that have more variables (voxels) than observations (trials). With this method, the estimates of the category means and covariances are shrunken toward zero using James–Stein shrinkage estimators as a way to ensure the estimability of the inverse covariance matrix and to reduce the mean squared error when used for out-of-sample prediction.

At training, SDA models were derived from matrices whose rows corresponded to observations from the training set (trial-wise beta coefficient images) and whose columns corresponded to variables (brain voxels). For recall-trained classification, these matrices included recall trials from all functional runs except for the run that contained the “test” trials, so that seven matrices were generated per analysis for each participant (each matrix excluded data from one functional run). For perception-trained classification, SDA models were derived from a single matrix per participant whose rows corresponded to perception trials from all seven functional runs. From each matrix, SDA generated one set of voxel weights per video from the stimulus set. To predict which video was being recalled, the dot product between a trial's voxel beta coefficients and each video's set of weights was calculated, resulting in one discriminant score per trial per video. The classifier assigned a trial to the video whose set of weights generated the largest discriminant score, that is, to the category that elicited the pattern of distributed activity most similar to that trial's activity.

Classifier performance was assessed based on the percentage of correctly identified trials, a categorical measure. As there were 11 videos in our set, chance level corresponded to 1 of 11 or 9.09%. We also used the classifier's discriminant scores as a continuous measure of evidence in favor of each video. Because of the relatively large number of videos in the set (11), such a graded measure of classifier output offered more fine-grained information about the strength of evidence in favor of each video.

To determine the “upper limit” of classification based on brain activity at perception, we also trained and tested a third SDA pattern classifier via leave-one-run-out cross-validation on perception trials (i.e., the classifier was trained and tested on independent subsets of perception trials). Mean classification accuracy for our sample of 19 participants was 81.37% (SD = 9.03%), which is well above chance (9.091%). Over the 21 repeated viewings, mean classification accuracy fluctuated between 74.24% and 89.95%, indicating that video-specific signal was robust and consistent at perception.

Searchlight Pattern Classification

To generate a regional map indicating where item-specific patterns of activity elicited at perception were reinstated during mental replay, we performed a searchlight analysis over the entire brain (Kriegeskorte, Goebel, & Bandettini, 2006). Classification accuracy was computed for each trial based on activity from voxels restricted to a local 6-mm-radius spherical neighborhood that surrounded a central voxel. This SDA classifier was trained on perception trials and tested on mental replay trials (perception-trained classifier). The sphere was moved around the entire brain, excluding voxels falling outside a functional brain mask, to create a whole-brain map of classification accuracy scores attributed to the voxel in the center of each sphere. For group-level inference, whole-brain accuracy maps were spatially normalized to MNI space and analyzed with a voxelwise one-sample t test. To protect against any nonspecific signal (e.g., correlated motion) that could spuriously aid classification, we also computed the average classification accuracy within a mask comprising voxels in white matter and cerebrospinal fluid (CSF). To create this mask, all participants' high-resolution MP-RAGE structural images were segmented into gray matter, white matter, and CSF. These segmented images were warped to MNI space and then averaged across participants, yielding a probability map for each tissue class. Voxels in MNI space with a CSF or white matter probability greater than .95 were included in the mask. The average classification accuracy within this mask was .1004, which is slightly above the chance accuracy of .09091 (1/11 videos). We used the former value as the baseline value (expected value under the null hypothesis) when we computed the voxelwise one-sample t test to identify significant classification accuracy at the group level. We report significant clusters (more than five voxels; t(18) > 3.93, p < .001 uncorrected, which corresponds to a whole-brain alpha < .05 corrected based on a Monte Carlo simulation conducted with AFNI's AlphaSim) indentified by the group analysis in Table 3 and display the results in Figure 5 (third row; t(18) > 3.20, p < .005 uncorrected). Throughout the Results section, we adopted more conservative thresholds for coordinate tables to safeguard against false-positives, but we adopted a more liberal threshold for figures because their format makes is possible to display a range of significance values.

LGA

We identified brain regions where high BOLD activity was monotonically related to high classifier evidence, irrespective of the particular video that was being recalled. For each participant, Spearman correlations were computed in each voxel between activity level (trial-wise beta coefficient) for individual recall trials and trial-specific classifier evidence (whole-brain SDA discriminant scores from the perception-trained classifier; see Pattern Classification). Separate correlation coefficients were computed for each video. These coefficient maps were then averaged over videos to derive the local–global correlation maps. Coefficient maps were spatially normalized to MNI space, and a group-level analysis was performed with a voxelwise one-sample t test. Significant clusters (more than five voxels; t(18) > 3.93, p < .001 uncorrected, p < .05 corrected with Monte Carlo simulation) are reported in Table 1 and displayed in Figure 4 (top; t(18) > 3.20, p < .005 uncorrected).

Table 1. 

Brain Regions Whose Activity during Mental Replay Correlated Significantly with Perception-trained Classification

Anatomical RegionHemiBACluster Sizet ScoreMNI Coordinates
xyz
Positive Correlations 
Middle occipital gyrus/cuneus 18 87 6.95 23.5 −98.5 9.5 
Inferior occipital gyrus 17 46 6.12 −18.5 −104 −5.5 
Precuneus/superior parietal lobule 39 6.06 −9.5 −77.5 48.5 
Precuneus/superior parietal lobule 29 5.07 23.5 −68.5 45.5 
Middle temporal gyrus 37 24 8.29 −54.5 −68.5 −2.5 
Precentral gyrus 12 7.42 −54.5 0.5 45.5 
Superior frontal gyrus 11 5.58 −27.5 −5.5 69.5 
SMA 11 5.49 −3.5 3.5 63.5 
Putamen n/a 11 4.87 23.5 9.5 6.5 
Putamen n/a 4.86 −21.5 9.5 6.5 
Intraparietal sulcus 40 4.75 −33.5 −44.5 33.5 
Caudate nucleus n/a 17.5 0.5 18.5 
Middle occipital gyrus 19 4.41 41.5 −83.5 15.5 
Hippocampusa n/a 6a 4.87 20.5 −20.5 −11.5 
 
Negative Correlations 
Medial frontal gyrus 10 369 7.45 5.5 57.5 −2.5 
Angular gyrus 39 102 9.71 47.5 −62.5 30.5 
Angular gyrus 39 39 5.59 −51.5 −56.5 30.5 
Posterior cingulate cortex 31 31 5.81 −0.5 −47.5 36.5 
Inferior frontal cortex (p. orbit) 47 31 5.92 −30.5 21.5 −17.5 
Middle frontal gyrus 30 6.18 −30.5 27.5 51.5 
Middle temporal gyrus 21 30 6.60 −63.5 −23.5 −14.5 
Superior frontal gyrus 21 6.25 23.5 36.5 54.5 
Fusiform gyrus 19 19 5.14 −27.5 −74.5 −8.5 
Middle frontal gyrus 18 7.55 44.5 15.5 54.5 
Superior frontal gyrus 10 14 5.02 17.5 63.5 21.5 
Middle temporal gyrus 21/22 12 5.04 59.5 −35.5 3.5 
Inferior frontal gyrus (p. orbit) 47 10 5.14 38.5 21.5 −20.5 
Middle cingulate cortex 31 9.10 5.5 −23.5 36.5 
Anterior cingulate cortex 24 4.55 2.5 30.5 −8.5 
Superior temporal gyrus 22 5.93 56.5 −8.5 −5.5 
Middle temporal gyrus 21 4.66 65.5 −17.5 −14.5 
Precuneus 5.88 −3.5 −47.5 51.5 
Anatomical RegionHemiBACluster Sizet ScoreMNI Coordinates
xyz
Positive Correlations 
Middle occipital gyrus/cuneus 18 87 6.95 23.5 −98.5 9.5 
Inferior occipital gyrus 17 46 6.12 −18.5 −104 −5.5 
Precuneus/superior parietal lobule 39 6.06 −9.5 −77.5 48.5 
Precuneus/superior parietal lobule 29 5.07 23.5 −68.5 45.5 
Middle temporal gyrus 37 24 8.29 −54.5 −68.5 −2.5 
Precentral gyrus 12 7.42 −54.5 0.5 45.5 
Superior frontal gyrus 11 5.58 −27.5 −5.5 69.5 
SMA 11 5.49 −3.5 3.5 63.5 
Putamen n/a 11 4.87 23.5 9.5 6.5 
Putamen n/a 4.86 −21.5 9.5 6.5 
Intraparietal sulcus 40 4.75 −33.5 −44.5 33.5 
Caudate nucleus n/a 17.5 0.5 18.5 
Middle occipital gyrus 19 4.41 41.5 −83.5 15.5 
Hippocampusa n/a 6a 4.87 20.5 −20.5 −11.5 
 
Negative Correlations 
Medial frontal gyrus 10 369 7.45 5.5 57.5 −2.5 
Angular gyrus 39 102 9.71 47.5 −62.5 30.5 
Angular gyrus 39 39 5.59 −51.5 −56.5 30.5 
Posterior cingulate cortex 31 31 5.81 −0.5 −47.5 36.5 
Inferior frontal cortex (p. orbit) 47 31 5.92 −30.5 21.5 −17.5 
Middle frontal gyrus 30 6.18 −30.5 27.5 51.5 
Middle temporal gyrus 21 30 6.60 −63.5 −23.5 −14.5 
Superior frontal gyrus 21 6.25 23.5 36.5 54.5 
Fusiform gyrus 19 19 5.14 −27.5 −74.5 −8.5 
Middle frontal gyrus 18 7.55 44.5 15.5 54.5 
Superior frontal gyrus 10 14 5.02 17.5 63.5 21.5 
Middle temporal gyrus 21/22 12 5.04 59.5 −35.5 3.5 
Inferior frontal gyrus (p. orbit) 47 10 5.14 38.5 21.5 −20.5 
Middle cingulate cortex 31 9.10 5.5 −23.5 36.5 
Anterior cingulate cortex 24 4.55 2.5 30.5 −8.5 
Superior temporal gyrus 22 5.93 56.5 −8.5 −5.5 
Middle temporal gyrus 21 4.66 65.5 −17.5 −14.5 
Precuneus 5.88 −3.5 −47.5 51.5 

Activations are significant at p < .001 (uncorrected; t > 3.93; cluster threshold > five voxels), which corresponds to a whole-brain alpha of <.05 based on a Monte Carlo simulation conducted with AFNI's AlphaSim. The cluster's Brodmann's area (BA), its size in voxels (Cluster Size), and the coordinates (MNI space, in millimeters) and t value of its peak voxel are provided. Hemi = hemisphere; n/a = non-applicable; p. orbit = pars orbitalis; L = left; R = right.

aHippocampal cluster size at p < .005 uncorrected, t > 3.20, α < 0.5, based on a Monte Carlo simulation conducted within a hippocampal mask (small volume correction).

Vividness Regression Analysis

To estimate the relationship between univariate activity and vividness, we computed, within each voxel, a multiple regression analysis that assessed whether vividness ratings modulated trial-by-trial fluctuations in activity level relative to the implicit baseline. Video was modeled as a factor of no interest to account for changes in vividness ratings that were video specific. The number of repetitions was also entered as a covariate to control for the influence of repeated retrieval on activity levels and vividness ratings (Figure 2, center). Thus, our model identified brain regions whose level of activity during mental replay was modulated by subjective vividness independently of repetition and video. Significant clusters (more than five voxels; t(18) > 3.93, p < .001 uncorrected, p < .05 corrected with Monte Carlo simulation) are reported in Table 2 and displayed in Figure 4 (center; t(18) > 3.20, p < .005 uncorrected). To determine whether regions modulated by vividness also contributed to the specificity of memory reactivation, we performed a conjunction analysis between the group-level t score maps from the LGA and the vividness analysis. We used AFNI's 3dcalc program to create an inclusive mask of group-level t maps for each analysis (t(18) > 3.20, p < .005, for each map; Figure 4, bottom). In addition, we also quantified the global similarity between these two maps by computing the spatial correlation (over voxels) between the two images.

Figure 2. 

Classifier performance (percent correct) is shown as a function of stimulus repetition (left) and vividness ratings (right); chance level corresponds to the x axis (1/11 or 9.1%). Mean vividness ratings are also shown as a function of repetition (center). Error bars correspond to SEM.

Figure 2. 

Classifier performance (percent correct) is shown as a function of stimulus repetition (left) and vividness ratings (right); chance level corresponds to the x axis (1/11 or 9.1%). Mean vividness ratings are also shown as a function of repetition (center). Error bars correspond to SEM.

Table 2. 

Brain Regions Whose Activity during Mental Replay Correlated with Vividness Ratings

Anatomical RegionHemiBACluster Sizet ScoreMNI Coordinates
xyz
Positive Correlations 
Caudate nucleus n/a 105 7.17 20.5 0.5 21.5 
Putamen n/a 77 7.45 −24.5 12.5 −2.5 
Superior parietal lobule 51 6.26 23.5 −65.5 54.5 
Lingual gyrus 17/18 15 5.56 17.5 −92.5 −11.5 
Precentral gyrus 11 4.74 53.5 −2.5 48.5 
Precuneus/superior parietal lobule 5.51 11.5 −71.5 60.5 
Precentral gyrus 5.22 −39.5 −2.5 27.5 
Precuneus/superior parietal lobule 4.29 −18.5 −71.5 54.5 
Inferior temporal gyrus 37 4.27 −54.5 −65.5 −8.5 
 
Negative Correlations 
Medial pFC 32 502 7.6 5.5 45.5 0.5 
Angular gyrus 39 152 6.87 47.5 −65.5 45.5 
Middle temporal gyrus 21 105 7.28 62.5 −20.5 −8.5 
Inferior frontal gyrus (p. triang.) 45 92 8.54 44.5 30.5 −2.5 
Posterior cingulate cortex 31 64 5.57 −6.5 −47.5 30.5 
Angular gyrus 39 63 8.53 −54.5 −65.5 30.5 
Insula 13 44 6.53 −30.5 18.5 −11.5 
Middle cingulate cortex 31 31 6.8 2.5 −26.5 45.5 
Middle temporal gyrus 21 22 6.03 −54.5 −38.5 −5.5 
Middle temporal gyrus 21 11 6.78 50.5 9.5 −35.5 
Anterior cingulate cortex 24/33 4.69 2.5 24.5 18.5 
Angular gyrus 39 6.11 62.5 −53.5 18.5 
Anterior cingulate cortex 32 5.84 −12.5 48.5 9.5 
Anatomical RegionHemiBACluster Sizet ScoreMNI Coordinates
xyz
Positive Correlations 
Caudate nucleus n/a 105 7.17 20.5 0.5 21.5 
Putamen n/a 77 7.45 −24.5 12.5 −2.5 
Superior parietal lobule 51 6.26 23.5 −65.5 54.5 
Lingual gyrus 17/18 15 5.56 17.5 −92.5 −11.5 
Precentral gyrus 11 4.74 53.5 −2.5 48.5 
Precuneus/superior parietal lobule 5.51 11.5 −71.5 60.5 
Precentral gyrus 5.22 −39.5 −2.5 27.5 
Precuneus/superior parietal lobule 4.29 −18.5 −71.5 54.5 
Inferior temporal gyrus 37 4.27 −54.5 −65.5 −8.5 
 
Negative Correlations 
Medial pFC 32 502 7.6 5.5 45.5 0.5 
Angular gyrus 39 152 6.87 47.5 −65.5 45.5 
Middle temporal gyrus 21 105 7.28 62.5 −20.5 −8.5 
Inferior frontal gyrus (p. triang.) 45 92 8.54 44.5 30.5 −2.5 
Posterior cingulate cortex 31 64 5.57 −6.5 −47.5 30.5 
Angular gyrus 39 63 8.53 −54.5 −65.5 30.5 
Insula 13 44 6.53 −30.5 18.5 −11.5 
Middle cingulate cortex 31 31 6.8 2.5 −26.5 45.5 
Middle temporal gyrus 21 22 6.03 −54.5 −38.5 −5.5 
Middle temporal gyrus 21 11 6.78 50.5 9.5 −35.5 
Anterior cingulate cortex 24/33 4.69 2.5 24.5 18.5 
Angular gyrus 39 6.11 62.5 −53.5 18.5 
Anterior cingulate cortex 32 5.84 −12.5 48.5 9.5 

Activations are significant at p < .001 (uncorrected; t > 3.93; cluster threshold > five voxels), which corresponds to a whole-brain alpha of <.05 based on a Monte Carlo simulation conducted with AFNI's AlphaSim. The cluster's Brodmann's area (BA), its size in voxels (Cluster Size), and the coordinates (MNI space, in millimeters) and t value of its peak voxel are provided. p. triang = pars triangularis.

Item and Participant Conjunction Analyses

Correlation coefficients averaged across videos reflect the average tendency for local activity in a given voxel to correlate with an output measure (the SDA discriminant score or vividness rating) regardless of the video recalled. However, to ensure that an average measure was not driven by one or two videos, we conducted group analyses over participants (i.e., the standard random effects analysis in neuroimaging) and an analysis over items, for both the LGA and the vividness analysis. Item analyses are commonly used in psycholinguistics (Clark, 1973) to ensure that any effect observed in a group of participants generalizes across the set of items used in a study (e.g., the particular words used in a lexical decision task), but they have also been used in neuroimaging (Bedny, Aguirre, & Thompson-Schill, 2007). Here, we employed an items-based analysis to ensure that any observed group effect in the LGA or vividness analysis generalizes beyond the set of videos used in our study.

For simplicity, we used parallel approaches for the LGA and the vividness analysis: For each participant, we computed separate Spearman voxelwise correlations per video between a voxel's activity level (trial-wise beta coefficient) and either trial-specific vividness ratings or whole-brain SDA discriminant scores. This approach yielded 11 correlation maps per participant for the vividness analysis and 11 maps for the LGA (one map per video). To generalize across participants, we averaged a participant's (vividness or LGA) correlation maps across videos and carried out voxelwise t tests on these 19 participant-specific mean correlation maps. To generalize across items, we averaged (vividness or LGA) correlation maps from the same video across participants, and we carried out voxelwise t tests on the 11 video-specific mean correlation maps averaged over participants.

To identify signal that generalized across both videos and participants, we generated inclusive conjunction masks by intersecting the two (items and participants) t test maps using AFNI's 3dcalc program. We generated inclusive item + participant masks based on two different thresholds, p < .01 (item: t(10) > 3.17, participant: t(18) > 2.88) and p < .005 (item: t(10) > 3.59, participant: t(18) > 3.20), with the more lenient threshold used for the purposes of visual display. Masks based on each threshold are superimposed in Figure 5 (LGA: top, vividness analysis: second row). We observed a great resemblance between regions displayed in Figures 4 and 5 for the LGA and the vividness analysis, respectively, indicating that effects picked up by these analyses truly generalized across videos.

Vividness DISTATIS Analysis

To visualize the relationship between brain activity patterns associated with memory for each video, we generated maps of representational space based on the perception-trained classifier's output. For each participant, SDA discriminant scores from the classifier were normalized (z scored) within each of the 11 prediction categories (the classifier's prediction of how likely a trial belonged to each video). Discriminant scores from each of the 11 prediction categories were averaged over trials for which the same video was retrieved, a process yielding an 11 × 11 matrix of average discriminant scores. Distance matrices were created by computing the Euclidean distance between pairs of rows from the 11 × 11 matrix, so that pairs of retrieved videos were compared along mean discriminant scores from the 11 prediction categories. These distance matrices were projected onto factor maps based on a three-way generalization of classical multidimensional scaling analysis called DISTATIS (Abdi, Williams, Valentin, & Bennani-Dosse, 2012; Abdi, Dunlop, & Williams, 2009; Abdi, 2007). In DISTATIS, multiple distance matrices are optimally integrated into a common distance matrix that is then analyzed with multidimensional scaling, so that distance matrices derived from multiple individuals can be combined. Here, distance matrices were decomposed into factor maps on which videos were represented as points, and the distance between points best approximated the classifier-derived distances. Videos whose discriminant scores tended to vary together were closer on the factor map than videos whose discriminant scores did not covary with one another. A bootstrap-based procedure (Abdi et al., 2009, 2012) was used to compute 95% confidence ellipsoid intervals around each video. When confidence ellipsoids did not intersect between pairs of videos, the two videos were considered significantly different at p < .05. Note again that, in this analysis, visualizations are based on the discriminant scores that were derived by projecting the recall trial beta images onto the perception-trained model solution with SDA.

To visualize the influence of vividness on classifier performance, we then assessed whether ratings modulated patterns of interpoint distances on the factor maps. We computed two distance matrices for each participant: one based on trials rated as highly vivid (“high”) and one based on trials that were given lower ratings (“low”). Within participants, trials for each video were categorized as high or low according to the following set of rules: First, the video's 21 trials were split into bins based on their vividness rating (1–4). Then, the four bins were recombined into two bins based on the solution that minimized the difference between the numbers of trials entered in each bin. This measure optimized the distribution of trials while accounting for variations in scale anchoring. If fewer than five trials were included in a bin (e.g., video 5's “low” bin), trials were reassigned from the larger bin (video 5's “high” bin) to the smaller bin until the latter contained five trials. Some trial reassignment was required in 34.45% of 209 cases (19 participants × 11 videos; one or two trials reassigned: 19.14% of cases; three or more trials reassigned: 15.31% of cases). The reassignment procedure reduced the influence of single trials in bins with too few trials. In the very rare case when all of a video's trials fell in a single rating bin, trials were split randomly between a “high” bin and a “low” bin (this situation only happened for a single video for a single participant). As a result, each cell was filled in the participant's data matrix so that distance matrices could be calculated. We projected the two distance matrices based on the subsets of high and low vividness trials onto the original DISTATIS factor map derived from all trials. We predicted that centroids should be projected further away from the origin for high than low vividness trials, indicating superior separation of the high vividness images and more distinct neural memory representations.

RESULTS

Whole-brain Pattern Classifier Performance

Both the recall-trained and perception-trained classifiers performed significantly above chance (1/11 or 9.1%) at all repetitions (Figure 2, left; one-sample t tests, t(17/18) ranging from 2.74 to 9.94, all ps < .05]. In addition, the recall-trained classifier outperformed the perception-trained classifier, as revealed by a two-way ANOVA with Rating and Repetition entered as within-participant factors (main effect of Classifier: F(1, 15) = 94.90, p < .001). This result was expected because the recall-trained classifier was trained and tested on trials from the same condition: This classifier could take advantage of all reliable video-specific patterns present during mental replay regardless of whether they were also present during perception. The perception-trained classifier, however, was a more stringent test of reactivation: It performed well only to the extent that patterns of information present during mental replay were also contained in the perception trials. There was no significant main effect of Repetition, F(20, 300) = 0.92, p = .57, or Repetition × Classifier interaction effect, F(20, 300) = 1.14, p = .305, on classification accuracy. In three participants, classification data were missing for a few repetitions: An EPI run was repeated, which caused a shift in the repetition numbers assigned to valid subsequent trials. Partial data from these three participants are included in the means plotted in Figure 2 but were excluded from the ANOVA.

The distribution of vividness scores indicates that most mental replay trials were performed successfully and given high ratings (on average, ratings of 1, 2, 3, and 4 were given to 1.58% [SD = 1.63%], 13.39% [SD = 12.59%], 45.36% [SD = 18.32%], and 38.75% [SD = 28.32%] of the trials, respectively). Mean vividness ratings increased within the first few repetitions and remained high throughout the task (Figure 2, center). Importantly, both the recall-trained and perception-trained classifiers' performance increased as a function of vividness ratings (Figure 2, right), as revealed by a two-way ANOVA with Rating and Classifier model (perception-trained, recall-trained) entered as within-participant factors (main effect of Classifier model: F(1, 12) = 21.36, p < .001; main effect of rating: F(3, 36) = 17.92, p < .001; linear contrast for the rating factor: F(1, 12) = 33.11, p < .001). The quadratic and cubic contrasts for ratings did not reach significance (p ≥ .065), and neither did the Rating × Classifier interaction: F(3, 36) = 1.50, p = .232; data from the six participants who did not make use of all four values on the 1–4 rating scale were excluded from the ANOVA but are included in the means plotted in Figure 2. Our results indicate that video-specific patterns of brain activity elicited during trials perceived as subjectively more vivid were objectively more distinct—as indicated by the recall-trained classifier's performance—and that a significant portion of this activity was modeled on perception—as indicated by the perception-trained classifier's performance.

The DISTATIS analysis based on the perception-trained classifier's performance provides an overall view of the videos' representation in “memory space.” The first dimension on the factor map (Figure 3A, x axis) explained 29% of the total variance in the distance matrix and opposed videos that included speech or music (e.g., a voiced-over commercial, Barack Obama giving a speech; right side) to videos that contained noise (e.g., glass breaking, a roller coaster sound; left side). The second dimension (Figure 3A, y axis) explained 18% of the variance and opposed videos that included biological motion (e.g., four ballet jazz dancers, a diver; bottom) to videos that either were more static or included nonbiological motion (e.g., a car, an avalanche, a building; top). The small amount of overlap between adjacent confidence intervals indicates that memory for most videos elicited distinctive patterns of cortical activation. Figure 3B shows the projections of each video onto Figure 3A's solution space based on distance matrices computed from trials with high (denoted “h”) and low (denoted “l”) vividness ratings, respectively (the convex hull is plotted for high and low vividness responses). For 9 of 11 videos in the stimulus set, trials low in vividness were closer to the origin than highly vivid trials (p = .033, by sign test), a pattern supported by the observation that the convex hull for vivid trials mostly included the convex hull for less vivid trials. These results confirm that the relationship between stimulus-specific neural reactivation and the vividness of recall is consistent across the stimulus set.

Figure 3. 

Projection of videos onto 2-D factor maps according to distance matrices derived from perception-trained classification. (A) Solution space based on all trials with bootstrap-derived 95% confidence ellipsoid intervals. (B) Projection of videos onto the main factor map as a function of vividness ratings. Trials that received high (denoted “h”) and low (denoted “l”) vividness ratings are plotted for each video and are contained within their respective convex hull (high vividness = dark gray, low vividness = pale gray). For most videos, highly vivid trials are further away from the origin than less vivid trials, indicating that patterns of neural reactivation are more distinctive for highly vivid trials.

Figure 3. 

Projection of videos onto 2-D factor maps according to distance matrices derived from perception-trained classification. (A) Solution space based on all trials with bootstrap-derived 95% confidence ellipsoid intervals. (B) Projection of videos onto the main factor map as a function of vividness ratings. Trials that received high (denoted “h”) and low (denoted “l”) vividness ratings are plotted for each video and are contained within their respective convex hull (high vividness = dark gray, low vividness = pale gray). For most videos, highly vivid trials are further away from the origin than less vivid trials, indicating that patterns of neural reactivation are more distinctive for highly vivid trials.

LGA

The LGA identified regions whose trial-by-trial fluctuation in activity during recall correlated with perception-trained classification irrespective of video content (Figures 4 and 5, top; Table 1). Similar sets of regions were identified by the initial LGA for which correlations were averaged across videos (Figure 4, Table 1) and by the additional LGA for which intervideo effects were modeled explicitly using a conjunction between the “item” and “participant” contrast maps (Figure 5). Regions that correlated positively with classifier performance included the fusiform gyrus, the occipital cortex, and the precuneus—regions known to play a role in visual imagery and visuospatial processes—and the superior parietal lobule—a region involved in the top–down modulation of attention (Dosenbach, Fair, Cohen, Schlaggar, & Petersen, 2008; Corbetta & Shulman, 2002). We also observed positive correlations within motor regions that included the lateral premotor cortex, SMA, and dorsal striatum (putamen; Figure 6). The contribution of these regions to specific reactivation might reflect the reconstruction of sequential information, as participants were asked to mentally replay videos from beginning to end. Positive correlations were also observed in the intraparietal sulcus whose activity can reflect graded memory strength (Hutchinson et al., 2014; Cabeza et al., 2011). At a reduced threshold adjusted for small volume correction (p < .005, more than five voxels; t(18) > 3.20, p < .05 based on a Monte Carlo simulation conducted within a hippocampal mask), voxels whose activity correlated positively with classifier performance were present in the right hippocampus (Table 1, Figure 6), a region known for its role in recollection.

Figure 4. 

Local–global and vividness analyses. Regions whose activity during mental replay correlated either positively (warm colors) or negatively (cold colors) with the perception-trained classifier's performance (“LGA”: top) and with vividness ratings (“Vividness”: center; |t(18)| > 3.20, p < .005). “Conjunction” (bottom): inclusive masking of brain voxels that correlated both with vividness ratings and with perception-trained classifier performance (each contrast: |t(18)| > 3.20, p < .005); voxel values that fell within the conjunction mask were averaged between the two maps for display purposes.

Figure 4. 

Local–global and vividness analyses. Regions whose activity during mental replay correlated either positively (warm colors) or negatively (cold colors) with the perception-trained classifier's performance (“LGA”: top) and with vividness ratings (“Vividness”: center; |t(18)| > 3.20, p < .005). “Conjunction” (bottom): inclusive masking of brain voxels that correlated both with vividness ratings and with perception-trained classifier performance (each contrast: |t(18)| > 3.20, p < .005); voxel values that fell within the conjunction mask were averaged between the two maps for display purposes.

Figure 5. 

Item + participant conjunction analyses, MVPA searchlight, and univariate contrast. (Top) Brain regions identified by the item + participant conjunction analyses whose activity correlated either with classifier performance (“LGA”: first row) or with vividness ratings (“Vividness”: second row). Negative correlations are in blue (dark blue: p < .005, pale blue: p < .01), and positive correlations are in red (red: p < .005, peach: p < .01). For both analyses, the thresholds used to generate the inclusive masks were t(10) > 3.17 (p < .01) and t(10) > 3.59 (p < .005) for the items' contrast and t(18) > 2.88 (p < .01) and t(18) > 3.20 (p < .005) for the participants' contrast. Third row: MVPA searchlight. Content-sensitive brain voxels whose activity was reactivated at mental replay, based on locally computed perception-trained pattern classification. Only positive values are displayed: t(18) > 3.20, p < .005. Fourth row: univariate analysis depicting regions whose mean level of activation was either increased (warm colors) or decreased (cold colors) in comparison with the implicit baseline during mental replay (t(18) > 3.20, p < .005).

Figure 5. 

Item + participant conjunction analyses, MVPA searchlight, and univariate contrast. (Top) Brain regions identified by the item + participant conjunction analyses whose activity correlated either with classifier performance (“LGA”: first row) or with vividness ratings (“Vividness”: second row). Negative correlations are in blue (dark blue: p < .005, pale blue: p < .01), and positive correlations are in red (red: p < .005, peach: p < .01). For both analyses, the thresholds used to generate the inclusive masks were t(10) > 3.17 (p < .01) and t(10) > 3.59 (p < .005) for the items' contrast and t(18) > 2.88 (p < .01) and t(18) > 3.20 (p < .005) for the participants' contrast. Third row: MVPA searchlight. Content-sensitive brain voxels whose activity was reactivated at mental replay, based on locally computed perception-trained pattern classification. Only positive values are displayed: t(18) > 3.20, p < .005. Fourth row: univariate analysis depicting regions whose mean level of activation was either increased (warm colors) or decreased (cold colors) in comparison with the implicit baseline during mental replay (t(18) > 3.20, p < .005).

Figure 6. 

Item + participant conjunction analyses in subcortical structures. Voxels included within an inclusive item + participant conjunction mask whose activity correlated either with classifier performance (“LGA”: left) or with vividness ratings (“Vividness”: right). At p < .005 (items' contrast: t(10) > 3.59, participants' contrast: t(18) > 3.20), activity correlated positively with classifier performance in the right hippocampus (top left) and with both classifier performance and vividness ratings in the bilateral striatum (bottom left and right, respectively). A trend for a positive correlation with vividness ratings was observed in the right hippocampus at a lowered threshold (top right; p < .01; items' contrast: t(10) > 3.17, participants' contrast: t(18) > 2.88).

Figure 6. 

Item + participant conjunction analyses in subcortical structures. Voxels included within an inclusive item + participant conjunction mask whose activity correlated either with classifier performance (“LGA”: left) or with vividness ratings (“Vividness”: right). At p < .005 (items' contrast: t(10) > 3.59, participants' contrast: t(18) > 3.20), activity correlated positively with classifier performance in the right hippocampus (top left) and with both classifier performance and vividness ratings in the bilateral striatum (bottom left and right, respectively). A trend for a positive correlation with vividness ratings was observed in the right hippocampus at a lowered threshold (top right; p < .01; items' contrast: t(10) > 3.17, participants' contrast: t(18) > 2.88).

We also identified several brain regions whose activity correlated negatively with classifier performance. Many of these regions belonged to the default mode network (DMN; Buckner, Andrews-Hanna, & Schacter, 2008; Raichle et al., 2001) and included the medial pFC and posterior cingulate cortex, the angular gyrus, and the lateral temporal cortex. This finding is perhaps surprising given evidence of the DMN's consistent activation during tasks of autobiographical memory retrieval (Cabeza & St Jacques, 2007; Svoboda, McKinnon, & Levine, 2006) and other internally oriented tasks that require visualization (Spreng, 2012; Spreng, Stevens, Chamberlain, Gilmore, & Schacter, 2010; Buckner & Carroll, 2007). However, a well-established literature has consistently demonstrated activation within the DMN during mind wandering (e.g., Christoff, Gordon, Smallwood, Smith, & Schooler, 2009). During our task, it appears that relatively greater levels of DMN engagement were observed during trials for which participants performed poorly, possibly because their mind wandered off the task. A univariate contrast indicated that DMN activity was generally reduced from baseline during our mental replay task (Figure 5, bottom). Superior frontal cortex activity also correlated negatively with classifier performance in our results.

Vividness Analysis

We identified brain regions whose trial-by-trial activity during mental replay was modulated by participants' subjective vividness ratings across video (Figure 4 [center] and Figure 5 [second row], center; Table 2). Similar sets of regions were identified by the initial analysis (Figure 4) and by the additional analysis for which both item and participant contrast maps were combined via conjunction analysis (Figure 5). Positive correlations with vividness were observed in the occipital pole, lingual gyrus, precuneus and superior parietal lobule, dorsal striatum, premotor cortex and SMA, intraparietal sulcus, and left ventrolateral pFC. Negative correlates included regions from the DMN such as the medial pFC, angular gyrus, posterior cingulate cortex, middle temporal gyrus, and dorsolateral pFC. Hippocampal activity did not correlate significantly with vividness ratings (a nonsignificant trend is shown in Figure 6), although others have reported correlations between hippocampal activity and subjective ratings of imagery and detail content (Andrews-Hanna, Reidler, Sepulcre, Poulin, & Buckner, 2010; Viard et al., 2007; Addis, Moscovitch, Crawley, & McAndrews, 2004).

We hypothesized that whole-brain reactivation (as measured with SDA discriminant scores) and the subjective sense of reexperiencing an event (as measured with vividness ratings) are related by a common cause and that regions whose univariate activity predicts one measure should also predict the other. The inclusive mask that contained voxels identified by both the vividness analysis and the LGA demonstrates a striking correspondence between the two sets of findings (Figure 4, bottom). To evaluate this correspondence formally, we calculated the Pearson correlation between the respective “participants' factor” t test maps from the vividness analysis and the LGA. The correlation, which was calculated across voxels located within an average gray matter mask (where mean gray matter probability over participants > .25), was .57, indicating considerable similarity between the two images.

The multiple regression analysis conducted to identify regions modulated by vividness ratings also assessed the influence of repetition on activity during mental replay, as both ratings and the number of repetitions were entered as covariates. We identified two clusters where activity increased significantly as a function of repetition; these clusters were in the ventromedial pFC (MNI: −0.5, 54.5, −8.5) and the left anterior temporal cortex (MNI: −54.5, −8.5, −17.5), respectively.

Searchlight Pattern Classification

We conducted a searchlight MVPA to identify clusters of brain voxels whose stimulus-specific activity patterns were reactivated during mental replay. These regions mainly included secondary sensory, motor, and associative cortices and excluded primary visual and auditory cortices (Figure 5, third row; Table 3). Whereas patterns identified by the local–global and vividness analyses generalized across stimuli, the MVPA searchlight identified content-sensitive signal that distinguished among videos from the set; thus, some of the structures identified by the searchlight likely underlie the representation of stimuli held in memory during mental replay.

Table 3. 

Brain Regions Whose Activity Was Reactivated during Mental Replay, as Identified by a Searchlight MVPA

Anatomical RegionHemiBAClu. Sizet ScoreMNI Coordinates
xyz
Middle temporal gyrus 22 7090 12.7 −54.5 −53.5 18.5 
Precuneus/superior parietal lobulea 7090a 10.1 −9.5 −65.5 57.5 
Retrosplenial cortexa 23/30 7090a 10.1 −12.5 −62.5 18.5 
Posterior cingulate cortexa 23/31 7090a 10.1 5.5 −59.5 24.5 
Retrosplenial cortexa 29/30 7090a 10.0 2.5 −50.5 15.5 
Middle temporal gyrus 19/37 1622 11.1 47.5 −71.5 21.5 
Middle temporal gyrusa 21 1622a 8.7 62.5 −47.5 12.5 
Superior temporal gyrusa 22 1622a 9.6 50.5 −32.5 3.5 
Fusiform gyrusa 37 1622a 9.0 38.5 −35.5 −14.5 
Hippocampusa n/a 1622a 8.9 35.5 −35.5 −8.5 
Angular gyrusa 39 1622a 8.7 56.5 −50.5 24.5 
Inferior frontal gyrus (p. triang.) 45 319 7.6 50.5 42.5 6.5 
Medial frontal cortex 10/32 285 7.2 2.5 54.5 −8.5 
Superior frontal cortexa 285a 6.5 −0.5 54.5 36.5 
Superior frontal/precentral gyrus 68 5.1 −27.5 −8.5 60.5 
Insula 13 33 6.7 −39.5 0.5 3.5 
Middle temporal gyrus/temporal pole 21/38 33 5.8 50.5 −5.5 −17.5 
Superior frontal cortex 31 7.6 32.5 −5.5 60.5 
SMA 19 5.3 −0.5 −2.5 66.5 
Calcarine sulcus 17 15 5.0 −0.5 −86.5 12.5 
Temporal pole 38 10 4.5 50.5 15.5 −17.5 
Anatomical RegionHemiBAClu. Sizet ScoreMNI Coordinates
xyz
Middle temporal gyrus 22 7090 12.7 −54.5 −53.5 18.5 
Precuneus/superior parietal lobulea 7090a 10.1 −9.5 −65.5 57.5 
Retrosplenial cortexa 23/30 7090a 10.1 −12.5 −62.5 18.5 
Posterior cingulate cortexa 23/31 7090a 10.1 5.5 −59.5 24.5 
Retrosplenial cortexa 29/30 7090a 10.0 2.5 −50.5 15.5 
Middle temporal gyrus 19/37 1622 11.1 47.5 −71.5 21.5 
Middle temporal gyrusa 21 1622a 8.7 62.5 −47.5 12.5 
Superior temporal gyrusa 22 1622a 9.6 50.5 −32.5 3.5 
Fusiform gyrusa 37 1622a 9.0 38.5 −35.5 −14.5 
Hippocampusa n/a 1622a 8.9 35.5 −35.5 −8.5 
Angular gyrusa 39 1622a 8.7 56.5 −50.5 24.5 
Inferior frontal gyrus (p. triang.) 45 319 7.6 50.5 42.5 6.5 
Medial frontal cortex 10/32 285 7.2 2.5 54.5 −8.5 
Superior frontal cortexa 285a 6.5 −0.5 54.5 36.5 
Superior frontal/precentral gyrus 68 5.1 −27.5 −8.5 60.5 
Insula 13 33 6.7 −39.5 0.5 3.5 
Middle temporal gyrus/temporal pole 21/38 33 5.8 50.5 −5.5 −17.5 
Superior frontal cortex 31 7.6 32.5 −5.5 60.5 
SMA 19 5.3 −0.5 −2.5 66.5 
Calcarine sulcus 17 15 5.0 −0.5 −86.5 12.5 
Temporal pole 38 10 4.5 50.5 15.5 −17.5 

All activations are significant at p < .001 (uncorrected; t > 3.93; cluster threshold > five voxels), which corresponds to a whole-brain alpha of <.05 based on a Monte Carlo simulation conducted with AFNI's AlphaSim. The cluster's Brodmann's area (BA) and the coordinates (MNI space, in millimeters) and t value of each cluster's peak voxel are provided.

aLocal maximum within a larger cluster.

A visual comparison of the LGA map with the searchlight map highlights how each approach provides a different kind of window into the neural dynamics of memory retrieval. For example, the searchlight analysis does not reveal consistent reactivation in the occipital pole—in fact, the occipital pole was deactivated from baseline in the univariate contrast (Figure 5, bottom)—but the LGA indicates that increase in occipital pole activity correlates with the strength of whole-brain reactivation patterns (Figures 4 and 5; Table 1). Important differences also emerge between the LGA and searchlight results within structures from the DMN. The searchlight analysis indicates that at least some stimulus-specific reactivation was evident in canonical DMN regions such as the posterior cingulate cortex, the angular gyrus (bordering on the middle temporal gyrus), and, to a lesser extent, the medial pFC. On the other hand, the LGA and vividness analysis indicated that activity in these regions correlated negatively with whole-brain reactivation and with vividness ratings. In other words, whole-brain classification accuracy was poorer when DMN regions where activated, although their locally multivariate activity patterns seemed to support stimulus representation during mental replay. We address this apparent discrepancy between results from the LGA, vividness, and searchlight analyses in the Discussion.

DISCUSSION

Summary of Findings

Our results indicate a clear correspondence between cortical reinstatement and recollection. First, both the perception-trained and recall-trained classifiers' accuracy increased with vividness ratings, indicating that the reactivation of video-specific activity patterns was accompanied by an increase in the richness of the memory experience. Second, we observed a substantial correspondence between brain regions whose activity correlated with vividness ratings and regions whose activity correlated with pattern classification during mental replay. These results reveal a common neural circuit that supports objectively measurable neural reactivation and subjectively experienced memory recall.

Although the relationship between the graded quality of distributed item-specific reactivation and the vividness of recollection is intuitive, little evidence of a link between the two phenomena can be found in the current literature. Most existing studies have characterized recollection as a phenomenon that is either present or absent: Pattern classification has been shown to be superior when a word is recognized based on recollection rather than familiarity (Johnson et al., 2009), when a picture is recognized rather than forgotten (Ritchey et al., 2013), or when source information (Gordon et al., 2013) and idiosyncratic details (Staresina et al., 2012; Kuhl et al., 2011) are remembered about paired stimuli. Although insightful, these studies do not address whether reactivation increases in a manner that reflects the graded quality of conscious memory retrieval. Only a few recent studies have sampled a range of “above-threshold” recollective experiences. These studies have shown that reactivation reflects the vividness of recall (Johnson et al, in press), the amount of information recollected about a stimulus' features (Wing et al., 2015), and the context in which that stimulus was encoded (Leiker & Johnson, 2014). Together with our own observations, these findings suggest that subjective indices of recollection and pattern reactivation are two facets of a common underlying set of neural memory processes.

The current study relied on well-remembered, complex, and multimodal stimuli that typically elicited some amount of mental imagery—a defining feature of recollection (Conway, 2009; Rubin et al., 2003; Brewer, 1986, 1995). In contrast to previous studies that have examined how measures such as vividness ratings relate to brain activity across items, we examined this relationship over multiple repetitions of the same item. This procedure ensured that correlations between univariate brain activity and vividness or whole-brain neural reactivation were unconfounded with differences in content or imageability between stimulus items. The repetition of items also made it possible to estimate whether effects were likely to generalize beyond stimuli from the experimental set. With this approach, we identified content-general processes that benefitted both cortical reinstatement and vivid mentalization during mental replay.

Positive Correlates of Reactivation and Vividness

Several regions identified by the LGA overlapped with regions whose univariate activity correlated with vividness ratings. Activity correlated positively with vividness and reactivation in motor regions that included the striatum (caudate nucleus and putamen), the SMA, and the precentral gyrus. The involvement of motor-related regions in our mental replay task may have reflected the sequential nature of our videos. For example, the striatum's involvement in learning and memory is well known, although this structure is traditionally thought to play a role in unconscious habit and stimulus–response learning rather than in conscious associative learning (Packard, 2010). However, the striatum is also thought to compress the representation of temporally ordered motor and cognitive action sequences and to be involved in the performance of learned sequences (Jin, Tecuapetla, & Costa, 2014; Yin, 2010; Schubotz & von Cramon, 2002; Graybiel, 1998). Although these forms of sequential learning are often implicit, evidence indicates that a modulation of striatal activity by medial prefrontal regions is observed during the conscious retrieval of explicitly learned sequences of action, indicating that implicit learning processes can be controlled by conscious processes (Destrebecqz et al., 2005). Unit-recording evidence from animals and human lesion studies also suggest that motor sequences are represented in the SMA and pre-SMA (Halsband, Matsuzaka, & Tanji, 1994; Tanji & Shima, 1994; Halsband, Ito, Tanji, & Freund, 1993). In the current study, “mentally replaying” videos required participants to form dynamic mental representations of image sequences. Others have shown that striatal activity at encoding is linked with good subsequent declarative memory for video content (Ben-Yakov & Dudai, 2011). Here, our results show that recalling videos engages a circuit of motor and premotor regions that contributes to the vividness and specificity with which dynamic mental representations are consciously reconstructed from long-term memory.

The LGA also identified positive correlations between whole-brain reactivation and univariate activity in a cluster of right hippocampal voxels. The importance of the hippocampus to recollection is well established (Ranganath, 2010; Yonelinas et al., 2010; Moscovitch et al., 2005), and the current results are consistent with its role as an index of memory content that contributes to specific reactivation during recall (Teyler & Rudy, 2007; Teyler & DiScenna, 1986). Our results are also in line with other reports of a relationship between hippocampal activation and either task-specific (faces vs. scenes; Gordon et al., 2013) or item-specific (visual scenes: Ritchey et al., 2013; word–scene pairings: Staresina et al., 2012) cortical reinstatement.

Other correlates of vividness and reactivation included the intraparietal sulcus and the superior parietal lobule, two structures involved in the top–down (goal-directed) modulation of attention (Hutchinson et al., 2014; Ciaramelli, Grady, & Moscovitch, 2008; Corbetta, Patel, & Shulman, 2008; Dosenbach et al., 2008; Corbetta & Shulman, 2002). Intraparietal sulcus activity has been shown to covary with graded item memory strength (Hutchinson et al., 2014; Johnson et al., 2013; Wagner et al., 2005). Activity in regions involved in visual processes, such as the precuneus (Cavanna & Trimble, 2006; Fletcher et al., 1995) and the occipital pole, also correlated significantly with perception-trained classification and with vividness ratings.

Interestingly, our searchlight MVPA indicated that content-specific patterns were not typically reactivated in early visual cortices but instead took place in higher order cortical regions (see also St-Laurent et al., 2014; Buchsbaum, Lemire-Rodger, Fang, & Abdi, 2012). On average, the occipital pole was deactivated during mental replay trials (Figure 5). In other words, low-level visual cortices were activated only during very vivid trials for which whole-brain reactivation was highly specific. Our inability to detect pattern-specific content indicates that occipital pole activity was generally associated with highly vivid trials irrespective of the particular video recalled (Pratte & Tong, 2014; Xing, Ledgeway, McGraw, & Schluppeck, 2013; Slotnick, Thompson, & Kosslyn, 2005; Kosslyn & Thompson, 2003). It is possible that top–down influences were driving low-level visual cortex activity during mental replay, using the region's retinotopic infrastructure as a scaffold onto which visualizations were projected. These visualizations may or may not have been modeled on perception. For example, it may be that mental images were not projected reliably into the same retinotopic frame in which they were processed during direct perception (but see Naselaris, Olman, Stansbury, Ugurbil, & Gallant, 2015). Our results provide some evidence of the visual cortex's participation in the reactivation of vivid memories, and future work should attempt to elucidate the nature of this contribution.

We hypothesized that LGA would identify content-general neural signal, whereas the searchlight MVPA would identify regions supporting content-specific representations. Although each analysis identified qualitatively different distributed neural patterns, several brain regions, for example, the superior parietal lobule, inferior occipital gyrus, precentral gyrus, angular gyrus, and dorsomedial pFC, were identified by both analyses (Figure 5). Thus, some regions contained activity that covaried with the strength of whole-brain reactivation in a content-general manner (LGA), but these regions also contained locally multivariate information that reliably discriminated between videos from the stimulus set (searchlight analysis). We should note, however, that interpreting patterns of overlap in this case is complicated by the fact that the searchlight analysis is performed over a spherical ROI whereas the LGA is performed within single voxels, and therefore, the two analyses are not fully commensurable. Nevertheless, it appears that regions showing overlap in these two analyses may contain both content-general and content-specific signals that are separately picked out by the two analyses. Alternatively, either the LGA or the searchlight analyses are not pure measures of content-general and content-specific information, and therefore, overlap between these two analyses is driven by a common underlying cause. However, this possibility is mitigated by our combining of both item and participant analyses for the LGA to ensure that univariate correlations with whole-brain reactivation measures generalized across videos.

Negative Correlates of Reactivation and Vividness

We identified a widespread set of regions where univariate activity correlated negatively with both vividness and whole-brain reactivation. Several of these regions, which included the medial pFC, angular gyrus, precuneus, and lateral temporal cortex, are consistent with areas typically observed in studies of the DMN (Buckner et al., 2008; Raichle et al., 2001). These findings were somewhat unexpected, as DMN regions tend to be positively activated during certain kinds of memory recall, for example, during tasks of autobiographical memory retrieval (Andrews-Hanna et al., 2010; Spreng, Mar, & Kim, 2009; Buckner & Carroll, 2007), and thus, we thought they might be recruited by our mental replay task. Instead, these DMN regions were significantly deactivated during mental replay (Figure 5), and their activation was associated with poor mental replay in both the LGA and vividness analysis.

Paradoxically, the MVPA searchlight revealed local activity that was content sensitive and reactivated during replay in several of these DMN regions including the posterior cingulate cortex, precuneus, angular gyrus, and lateral temporal cortex. This finding echoes other reports of local reactivation patterns within DMN regions during the free recall of movie scenes (Chen, Leong, & Hasson, 2014). Thus, our results indicate (1) that DMN regions were deactivated during mental replay (i.e., main effect over videos was negative) and (2) that this deactivation generically benefitted pattern classification across videos, but also (3) that some DMN regions contained locally multivariate information that distinguished between different videos. A few things need to be noted to reconcile these findings. First, it is possible for MVPA to classify items or conditions successfully based on deactivation patterns that nevertheless contain content-sensitive information. In our case, DMN regions may have been deactivated below baseline during our task in a manner that differed across videos and therefore was sufficient to achieve significant local classification. Whether this video-specific signal variation was related to content representation per se or whether it reflected variations in the deployment of domain-general processes that consistently varied across videos is unclear, and further work is clearly needed to decide between these possibilities.

Second, our results may reflect the heterogeneity of function in the inferior parietal cortex (IPL; Hutchinson et al., 2014)—a region frequently involved in recollection—as well as in other DMN areas mentioned above. For example, it has been suggested that activity in the supramarginal portion of the IPL reflects an attentional state that either favors (Quamme et al., 2010) or results from (Cabeza, 2008) recollection. It has also been suggested that the IPL (especially the angular gyrus) acts as an episodic buffer that supports memory representation at recall (Kuhl & Chun, 2014; Leiker & Johnson, 2014; Vilberg & Rugg, 2008, 2012; but see Johnson et al., 2013; Cabeza, Ciaramelli, & Moscovitch, 2012). In our results, univariate activity in the dorsal portion of the angular gyrus was anticorrelated with whole-brain reactivation, whereas its ventral and anterior portions contained locally multivariate video-specific signal as identified in the searchlight analysis. The presence of content-sensitive signal in the latter area is consistent with the view that the IPL actively represents retrieved memories as a kind of episodic buffer (Kuhl & Chun, 2014; Leiker & Johnson, 2014; Vilberg & Rugg, 2009). For this interpretation to hold, however—because of the constraints of our cross-decoding approach—the episodic buffer must contain similar activation patterns during perception and recall.

On the other hand, the content-general signal detected in the vividness analysis and LGA can be reconciled with attentional accounts of ventral parietal cortical function. According to the attention to memory model (Cabeza et al., 2011; Cabeza, Ciaramelli, Olson, & Moscovitch, 2008; Ciaramelli et al., 2008), when a memory is recovered, attention is redirected toward the newly retrieved memory by the ventral parietal cortex, which acts as an attentional circuit breaker. In this scenario, recollection is experienced in conjunction with ventral parietal activity. During our task, however, participants retrieved strong memories that were well rehearsed, and it is likely that search processes and retrieval fluency (e.g., how easily or spontaneously a memory could be accessed) contributed minimally to trial-to-trial fluctuations in vividness and accuracy. Instead, participants were required to repeatedly reconstruct and visualize a complex collection of memory details, and it is likely that successful mental replay benefitted heavily from top–down processes (e.g., concentration, active mentalization, working memory). The interruption of top–down processes may have therefore interfered with the quality of recollection because attention was redirected away from mental replay, possibly because of distraction such as task-independent thought. Thus, activation increased in the angular gyrus when mental replay was poor. This interpretation is consistent with evidence that DMN becomes engaged during mind-wandering (Christoff et al., 2009): During trials with low vividness ratings, participants may have engaged in naturalistic mind-wandering, which may have engaged DMN regions to a greater extent than our mental replay task (Andrews-Hanna, Saxe, & Yarkoni, 2014; Spreng, 2012; Spreng & Grady, 2010; Christoff et al., 2009; Spreng et al., 2009; Buckner & Carroll, 2007). Because of the peculiarities of our task, an entirely different activation profile emerges that supports vivid mental replay and reactivation, providing interesting insight into the neural mechanisms of recollection.

Limitations

It must be noted that stimuli were shown multiple times during our task, and so the memories cued by our paradigm were “repisodic” or “generic” (Conway & Pleydell-Pearce, 2000; Brewer, 1986; Neisser, 1981) rather than purely episodic because of their nonspecific spatiotemporal context. However, studies in which participants imagine events in the future or visualize themselves in a novel context have shown that vivid and detailed mental constructs that lack a specific temporal context can elicit a sense of ecphory (aka mental time travel; Tulving, 1984, 2002; Suddendorf & Corballis, 1997; Wheeler, Stuss, & Tulving, 1997) and do engage similar neural circuits as memory for single past episodes (Schacter et al., 2012; Kwan, Carson, Addis, & Rosenbaum, 2010; Nyberg, Kim, Habib, Levine, & Tulving, 2010; Spreng & Grady, 2010; Hassabis & Maguire, 2009; Buckner & Carroll, 2007). In addition, memory for events that take place repeatedly in a consistent context shares several of the features that define episodic memory (Renoult, Davidson, Palombo, Moscovitch, & Levine, 2012), including their level of detail and visuospatial imagery content (St-Laurent, Moscovitch, Levine, & McAndrews, 2009) and the potential to give rise to a recollective experience (Brewer, 1986, 1995; but see Conway & Pleydell-Pearce, 2000). The pattern of neural activity elicited by memory for repeated events also shares many similarities with single memory episodes, and both event types are associated with hippocampal activation (Addis et al., 2004; although see Holland, Addis, & Kensinger, 2011, for some distinctions). In other words, the type of memory assessed in the current study had several—but not all—of episodic memory's defining features, including the capacity to evoke a subjective sense of revisiting the past in one's mind. Although our paradigm was adequate to capture the neural mechanisms of the recollective experience, it was not a “pure” test of episodic memory.

Conclusion

We combined multivariate and univariate analyses to identify content-general and content-specific neural mechanisms that contribute to specific stimulus representation in long-term memory. We successfully identified subsets of brain regions that supported both the reactivation of multimodal memory content and the vividness of the memory experience. In doing so, we provided evidence that whole-brain MVPA can be used as an external and objective marker of the graded quality of recollection for complex episodes and more generally as an indicator of memory specificity. This particular finding is especially relevant to research protocols for which reports of recollection are difficult to collect as well as to studies conducted in special populations whose subjective memory reports are unreliable, such as young children and patients with neurological conditions. Approaches that combine multivariate and univariate analyses can provide greater insights into the neural dynamics of cognition than either technique used on its own, and future applications could advance our understanding of the neural circuitry that supports a variety of mental constructs, from percepts to spatial maps to words and concepts.

Acknowledgments

This work was supported by a grant from the Canadian Institutes of Health Research (CIHR 106501 to B. R. B.), by a fellowship from the Katz Foundation (to M. S. L.), and by a visiting scientist fellowship from the Katz Foundation (to H. A.). The authors would like to thank Oles Chepesiuk, John Paul Koning, Ashley Bondad, and Sabrina Lemire-Rodger for their assistance.

Reprint requests should be sent to Dr. Marie St-Laurent, 3560 Bathurst Street, Toronto, Ontario, Canada M6A 2E1, or via e-mail: mstlaurent@research.baycrest.org.

REFERENCES

Abdi
,
H.
(
2007
).
Metric multidimensional scaling
. In
N. J.
Salkind
(Ed.),
Encyclopedia of measurement and statistics
.
Thousand Oaks, CA
:
Sage
.
Abdi
,
H.
,
Dunlop
,
J. P.
, &
Williams
,
L. J.
(
2009
).
How to compute reliability estimates and display confidence and tolerance intervals for pattern classifiers using the Bootstrap and 3-way multidimensional scaling (DISTATIS)
.
Neuroimage
,
45
,
89
95
.
Abdi
,
H.
,
Williams
,
L. J.
,
Valentin
,
D.
, &
Bennani-Dosse
,
M.
(
2012
).
STATIS and DISTATIS: Optimum multitable principal component analysis and three way metric multidimensional scaling
.
Wiley Interdisciplinary Reviews: Computational Statistics
,
4
,
124
167
.
Addis
,
D. R.
,
Moscovitch
,
M.
,
Crawley
,
A. P.
, &
McAndrews
,
M. P.
(
2004
).
Recollective qualities modulate hippocampal activation during autobiographical memory retrieval
.
Hippocampus
,
14
,
752
762
.
Andrews-Hanna
,
J. R.
,
Reidler
,
J. S.
,
Sepulcre
,
J.
,
Poulin
,
R.
, &
Buckner
,
R. L.
(
2010
).
Functional-anatomic fractionation of the brain's default network
.
Neuron
,
65
,
550
562
.
Andrews-Hanna
,
J. R.
,
Saxe
,
R.
, &
Yarkoni
,
T.
(
2014
).
Contributions of episodic retrieval and mentalizing to autobiographical thought: Evidence from functional neuroimaging, resting-state connectivity, and fMRI meta-analyses
.
Neuroimage
,
91C
,
324
335
.
Avants
,
B. B.
,
Epstein
,
C. L.
,
Grossman
,
M.
, &
Gee
,
J. C.
(
2008
).
Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain
.
Medical Image Analysis
,
12
,
26
41
.
Bedny
,
M.
,
Aguirre
,
G. K.
, &
Thompson-Schill
,
S. L.
(
2007
).
Item analysis in functional magnetic resonance imaging
.
Neuroimage
,
35
,
1093
1102
.
Ben-Yakov
,
A.
, &
Dudai
,
Y.
(
2011
).
Constructing realistic engrams: Poststimulus activity of hippocampus and dorsal striatum predicts subsequent episodic memory
.
Journal of Neuroscience
,
31
,
9032
9042
.
Bosch
,
S. E.
,
Jehee
,
J. F.
,
Fernandez
,
G.
, &
Doeller
,
C. F.
(
2014
).
Reinstatement of associative memories in early visual cortex is signaled by the hippocampus
.
Journal of Neuroscience
,
34
,
7493
7500
.
Brewer
,
W. F.
(
1986
).
What is autobiographical memory?
In
D. C.
Rubin
(Ed.),
Autobiographical memory
(pp.
25
49
).
Cambridge
:
Cambridge University Press
.
Brewer
,
W. F.
(
1995
).
What is recollective memory?
In
D. C.
Rubin
(Ed.),
Remembering our past: Studies in autobiographical memory
(pp.
19
66
).
Cambridge
:
Cambridge University Press
.
Buchsbaum
,
B. R.
,
Lemire-Rodger
,
S.
,
Fang
,
C.
, &
Abdi
,
H.
(
2012
).
The neural basis of vivid memory is patterned on perception
.
Journal of Cognitive Neuroscience
,
24
,
1867
1883
.
Buckner
,
R. L.
,
Andrews-Hanna
,
J. R.
, &
Schacter
,
D. L.
(
2008
).
The brain's default network: Anatomy, function, and relevance to disease
.
Annals of the New York Academy of Sciences
,
1124
,
1
38
.
Buckner
,
R. L.
, &
Carroll
,
D. C.
(
2007
).
Self-projection and the brain
.
Trends in Cognitive Sciences
,
11
,
49
57
.
Cabeza
,
R.
(
2008
).
Role of parietal regions in episodic memory retrieval: The dual attentional processes hypothesis
.
Neuropsychologia
,
46
,
1813
1827
.
Cabeza
,
R.
,
Ciaramelli
,
E.
, &
Moscovitch
,
M.
(
2012
).
Cognitive contributions of the ventral parietal cortex: An integrative theoretical account
.
Trends in Cognitive Sciences
,
16
,
338
352
.
Cabeza
,
R.
,
Ciaramelli
,
E.
,
Olson
,
I. R.
, &
Moscovitch
,
M.
(
2008
).
The parietal cortex and episodic memory: An attentional account
.
Nature Reviews Neuroscience
,
9
,
613
625
.
Cabeza
,
R.
,
Mazuz
,
Y. S.
,
Stokes
,
J.
,
Kragel
,
J. E.
,
Woldorff
,
M. G.
,
Ciaramelli
,
E.
, et al
(
2011
).
Overlapping parietal activity in memory and perception: Evidence for the attention to memory model
.
Journal of Cognitive Neuroscience
,
23
,
3209
3217
.
Cabeza
,
R.
, &
St Jacques
,
P.
(
2007
).
Functional neuroimaging of autobiographical memory
.
Trends in Cognitive Sciences
,
11
,
219
227
.
Cavanna
,
A. E.
, &
Trimble
,
M. R.
(
2006
).
The precuneus: A review of its functional anatomy and behavioural correlates
.
Brain
,
129
,
564
583
.
Chen
,
J.
,
Leong
,
Y. C.
, &
Hasson
,
U.
(
2014
).
Reinstatement of neural patterns during narrative free recall
.
Paper presented at the Annual Meeting of the Cognitive Neuroscience Society. Boston, MA
.
Christoff
,
K.
,
Gordon
,
A. M.
,
Smallwood
,
J.
,
Smith
,
R.
, &
Schooler
,
J. W.
(
2009
).
Experience sampling during fMRI reveals default network and executive system contributions to mind wandering
.
Proceedings of the National Academy of Sciences, U.S.A.
,
106
,
8719
8724
.
Ciaramelli
,
E.
,
Grady
,
C. L.
, &
Moscovitch
,
M.
(
2008
).
Top–down and bottom–up attention to memory: A hypothesis (AtoM) on the role of the posterior parietal cortex in memory retrieval
.
Neuropsychologia
,
46
,
1828
1851
.
Clark
,
H. H.
(
1973
).
The language-as-fixed-effect fallacy: A critique of language statistics in psychological research
.
Journal of Verbal Learning and Verbal Behavior
,
12
,
335
359
.
Conway
,
M. A.
(
2009
).
Episodic memories
.
Neuropsychologia
,
47
,
2305
2313
.
Conway
,
M. A.
, &
Pleydell-Pearce
,
C. W.
(
2000
).
The construction of autobiographical memories in the self-memory system
.
Psychological Review
,
107
,
261
288
.
Corbetta
,
M.
,
Patel
,
G.
, &
Shulman
,
G. L.
(
2008
).
The reorienting system of the human brain: From environment to theory of mind
.
Neuron
,
58
,
306
324
.
Corbetta
,
M.
, &
Shulman
,
G. L.
(
2002
).
Control of goal-directed and stimulus-driven attention in the brain
.
Nature Reviews Neuroscience
,
3
,
201
215
.
Cox
,
R. W.
(
1996
).
AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages
.
Computers and Biomedical Research
,
29
,
162
173
.
Danker
,
J. F.
, &
Anderson
,
J. R.
(
2010
).
The ghosts of brain states past: Remembering reactivates the brain regions engaged during encoding
.
Psychological Bulletin
,
136
,
87
102
.
Destrebecqz
,
A.
,
Peigneux
,
P.
,
Laureys
,
S.
,
Degueldre
,
C.
,
Del Fiore
,
G.
,
Aerts
,
J.
, et al
(
2005
).
The neural correlates of implicit and explicit sequence learning: Interacting networks revealed by the process dissociation procedure
.
Learning and Memory
,
12
,
480
490
.
Dosenbach
,
N. U.
,
Fair
,
D. A.
,
Cohen
,
A. L.
,
Schlaggar
,
B. L.
, &
Petersen
,
S. E.
(
2008
).
A dual-networks architecture of top–down control
.
Trends in Cognitive Sciences
,
12
,
99
105
.
Fletcher
,
P. C.
,
Frith
,
C. D.
,
Baker
,
S. C.
,
Shallice
,
T.
,
Frackowiak
,
R. S.
, &
Dolan
,
R. J.
(
1995
).
The mind's eye—Precuneus activation in memory-related imagery
.
Neuroimage
,
2
,
195
200
.
Gordon
,
A. M.
,
Rissman
,
J.
,
Kiani
,
R.
, &
Wagner
,
A. D.
(
2013
).
Cortical reinstatement mediates the relationship between content-specific encoding activity and subsequent recollection decisions
.
Cerebral Cortex
,
24
,
3350
3364
.
Graybiel
,
A. M.
(
1998
).
The basal ganglia and chunking of action repertoires
.
Neurobiology of Learning and Memory
,
70
,
119
136
.
Halsband
,
U.
,
Ito
,
N.
,
Tanji
,
J.
, &
Freund
,
H. J.
(
1993
).
The role of premotor cortex and the supplementary motor area in the temporal control of movement in man
.
Brain
,
116
,
243
266
.
Halsband
,
U.
,
Matsuzaka
,
Y.
, &
Tanji
,
J.
(
1994
).
Neuronal activity in the primate supplementary, pre-supplementary and premotor cortex during externally and internally instructed sequential movements
.
Neuroscience Research
,
20
,
149
155
.
Hassabis
,
D.
, &
Maguire
,
E. A.
(
2009
).
The construction system of the brain
.
Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences
,
364
,
1263
1271
.
Haxby
,
J. V.
(
2012
).
Multivariate pattern analysis of fMRI: The early beginnings
.
Neuroimage
,
62
,
852
855
.
Haynes
,
J. D.
, &
Rees
,
G.
(
2006
).
Decoding mental states from brain activity in humans
.
Nature Reviews Neuroscience
,
7
,
523
534
.
Holland
,
A. C.
,
Addis
,
D. R.
, &
Kensinger
,
E. A.
(
2011
).
The neural correlates of specific versus general autobiographical memory construction and elaboration
.
Neuropsychologia
,
49
,
3164
3177
.
Hutchinson
,
J. B.
,
Uncapher
,
M. R.
, &
Wagner
,
A. D.
(
2009
).
Posterior parietal cortex and episodic retrieval: Convergent and divergent effects of attention and memory
.
Learning and Memory
,
16
,
343
356
.
Hutchinson
,
J. B.
,
Uncapher
,
M. R.
,
Weiner
,
K. S.
,
Bressler
,
D. W.
,
Silver
,
M. A.
,
Preston
,
A. R.
, et al
(
2014
).
Functional heterogeneity in posterior parietal cortex across attention and episodic memory retrieval
.
Cerebral Cortex
,
24
,
49
66
.
Jin
,
X.
,
Tecuapetla
,
F.
, &
Costa
,
R. M.
(
2014
).
Basal ganglia subcircuits distinctively encode the parsing and concatenation of action sequences
.
Nature Neuroscience
,
17
,
423
430
.
Johnson
,
J. D.
,
McDuff
,
S. G.
,
Rugg
,
M. D.
, &
Norman
,
K. A.
(
2009
).
Recollection, familiarity, and cortical reinstatement: A multivoxel pattern analysis
.
Neuron
,
63
,
697
708
.
Johnson
,
J. D.
, &
Rugg
,
M. D.
(
2007
).
Recollection and the reinstatement of encoding-related cortical activity
.
Cerebral Cortex
,
17
,
2507
2515
.
Johnson
,
J. D.
,
Suzuki
,
M.
, &
Rugg
,
M. D.
(
2013
).
Recollection, familiarity, and content-sensitivity in lateral parietal cortex: A high-resolution fMRI study
.
Frontiers in Human Neuroscience
,
7
,
219
.
Johnson
,
M. K.
,
Kuhl
,
B. A.
,
Mitchell
,
K. J.
,
Ankudowich
,
E.
, &
Durbin
,
K. A.
(
in press
).
Age-related differences in the neural basis of the subjective vividness of memories: Evidence from multivoxel pattern classification
.
Cognitive, Affectective & Behavioral Neuroscience
.
Kosslyn
,
S.
, &
Thompson
,
W. L.
(
2003
).
When is early visual cortex activated during visual mental imagery?
Psychological Bulletin
,
129
,
723
746
.
Kriegeskorte
,
N.
(
2011
).
Pattern-information analysis: From stimulus decoding to computational-model testing
.
Neuroimage
,
56
,
411
421
.
Kriegeskorte
,
N.
,
Goebel
,
R.
, &
Bandettini
,
P.
(
2006
).
Information-based functional brain mapping
.
Proceedings of the National Academy of Sciences, U.S.A.
,
103
,
3863
3868
.
Kuhl
,
B. A.
, &
Chun
,
M. M.
(
2014
).
Successful remembering elicits event-specific activity patterns in lateral parietal cortex
.
Journal of Neuroscience
,
34
,
8051
8060
.
Kuhl
,
B. A.
,
Rissman
,
J.
,
Chun
,
M. M.
, &
Wagner
,
A. D.
(
2011
).
Fidelity of neural reactivation reveals competition between memories
.
Proceedings of the National Academy of Sciences, U.S.A.
,
108
,
5903
5908
.
Kwan
,
D.
,
Carson
,
N.
,
Addis
,
D. R.
, &
Rosenbaum
,
R. S.
(
2010
).
Deficits in past remembering extend to future imagining in a case of developmental amnesia
.
Neuropsychologia
,
48
,
3179
3186
.
Leiker
,
E. K.
, &
Johnson
,
J. D.
(
2014
).
Neural reinstatement and the amount of information recollected
.
Brain Research
,
1582
,
125
138
.
Li
,
S.
,
Mayhew
,
S. D.
, &
Kourtzi
,
Z.
(
2009
).
Learning shapes the representation of behavioral choice in the human brain
.
Neuron
,
62
,
441
452
.
Mahmoudi
,
A.
,
Takerkart
,
S.
,
Regragui
,
F.
,
Boussaoud
,
D.
, &
Brovelli
,
A.
(
2012
).
Multivoxel pattern analysis for fMRI data: A review
.
Computational and Mathematical Methods in Medicine
,
2012
,
961257
.
McDuff
,
S. G.
,
Frankel
,
H. C.
, &
Norman
,
K. A.
(
2009
).
Multivoxel pattern analysis reveals increased memory targeting and reduced use of retrieved details during single-agenda source monitoring
.
Journal of Neuroscience
,
29
,
508
516
.
Moscovitch
,
M.
,
Rosenbaum
,
R. S.
,
Gilboa
,
A.
,
Addis
,
D. R.
,
Westmacott
,
R.
,
Grady
,
C.
, et al
(
2005
).
Functional neuroanatomy of remote episodic, semantic and spatial memory: A unified account based on multiple trace theory
.
Journal of Anatomy
,
207
,
35
66
.
Naselaris
,
T.
,
Olman
,
C. A.
,
Stansbury
,
D. E.
,
Ugurbil
,
K.
, &
Gallant
,
J. L.
(
2015
).
A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes
.
Neuroimage
,
105
,
215
228
.
Neisser
,
U.
(
1981
).
John Dean's memory: A case study
.
Cognition
,
9
,
1
22
.
Norman
,
K. A.
,
Polyn
,
S. M.
,
Detre
,
G. J.
, &
Haxby
,
J. V.
(
2006
).
Beyond mind-reading: Multi-voxel pattern analysis of fMRI data
.
Trends in Cognitive Sciences
,
10
,
424
430
.
Nyberg
,
L.
,
Kim
,
A. S.
,
Habib
,
R.
,
Levine
,
B.
, &
Tulving
,
E.
(
2010
).
Consciousness of subjective time in the brain
.
Proceedings of the National Academy of Sciences, U.S.A.
,
107
,
22356
22359
.
Packard
,
M. G.
(
2010
).
Role of basal ganglia in habit learning and memory: Rats, monkeys and humans
. In
H.
Steiner
&
K. Y.
Tseng
(Eds.),
Handbook of basal ganglia structure and function
(pp.
561
569
).
New York
:
Academic Press
.
Pratte
,
M. S.
, &
Tong
,
F.
(
2014
).
Spatial specificity of working memory representations in the early visual cortex
.
Journal of Vision
,
14
,
22
.
Quamme
,
J. R.
,
Weiss
,
D. J.
, &
Norman
,
K. A.
(
2010
).
Listening for recollection: A multi-voxel pattern analysis of recognition memory retrieval strategies
.
Frontiers in Human Neuroscience
,
4
,
61
.
Raichle
,
M. E.
,
MacLeod
,
A. M.
,
Snyder
,
A. Z.
,
Powers
,
W. J.
,
Gusnard
,
D. A.
, &
Shulman
,
G. L.
(
2001
).
A default mode of brain function
.
Proceedings of the National Academy of Sciences, U.S.A.
,
98
,
676
682
.
Ranganath
,
C.
(
2010
).
A unified framework for the functional organization of the medial temporal lobes and the phenomenology of episodic memory
.
Hippocampus
,
20
,
1263
1290
.
Renoult
,
L.
,
Davidson
,
P. S.
,
Palombo
,
D. J.
,
Moscovitch
,
M.
, &
Levine
,
B.
(
2012
).
Personal semantics: At the crossroads of semantic and episodic memory
.
Trends in Cognitive Sciences
,
16
,
550
558
.
Rissman
,
J.
,
Gazzaley
,
A.
, &
D'Esposito
,
M.
(
2004
).
Measuring functional connectivity during distinct stages of a cognitive task
.
Neuroimage
,
23
,
752
763
.
Rissman
,
J.
, &
Wagner
,
A. D.
(
2012
).
Distributed representations in memory: Insights from functional brain imaging
.
Annual Review of Psychology
,
63
,
101
128
.
Ritchey
,
M.
,
Wing
,
E. A.
,
Labar
,
K. S.
, &
Cabeza
,
R.
(
2013
).
Neural similarity between encoding and retrieval is related to memory via hippocampal interactions
.
Cerebral Cortex
,
23
,
2818
2828
.
Rubin
,
D. C.
,
Schrauf
,
R. W.
, &
Greenberg
,
D. L.
(
2003
).
Belief and recollection of autobiographical memories
.
Memory & Cognition
,
31
,
887
901
.
Rugg
,
M. D.
,
Johnson
,
J. D.
,
Park
,
H.
, &
Uncapher
,
M. R.
(
2008
).
Encoding-retrieval overlap in human episodic memory: A functional neuroimaging perspective
.
Progress in Brain Research
,
169
,
339
352
.
Schacter
,
D. L.
,
Addis
,
D. R.
,
Hassabis
,
D.
,
Martin
,
V. C.
,
Spreng
,
R. N.
, &
Szpunar
,
K. K.
(
2012
).
The future of memory: Remembering, imagining, and the brain
.
Neuron
,
76
,
677
694
.
Schubotz
,
R. I.
, &
von Cramon
,
D. Y.
(
2002
).
Predicting perceptual events activates corresponding motor schemes in lateral premotor cortex: An fMRI study
.
Neuroimage
,
15
,
787
796
.
Slotnick
,
S. D.
,
Thompson
,
W. L.
, &
Kosslyn
,
S. M.
(
2005
).
Visual mental imagery induces retinotopically organized activation of early visual areas
.
Cerebral Cortex
,
15
,
1570
1583
.
Spreng
,
R. N.
(
2012
).
The fallacy of a “task-negative” network
.
Frontiers in Psychology
,
3
,
145
.
Spreng
,
R. N.
, &
Grady
,
C. L.
(
2010
).
Patterns of brain activity supporting autobiographical memory, prospection, and theory of mind, and their relationship to the default mode network
.
Journal of Cognitive Neuroscience
,
22
,
1112
1123
.
Spreng
,
R. N.
,
Mar
,
R. A.
, &
Kim
,
A. S.
(
2009
).
The common neural basis of autobiographical memory, prospection, navigation, theory of mind, and the default mode: A quantitative meta-analysis
.
Journal of Cognitive Neuroscience
,
21
,
489
510
.
Spreng
,
R. N.
,
Stevens
,
W. D.
,
Chamberlain
,
J. P.
,
Gilmore
,
A. W.
, &
Schacter
,
D. L.
(
2010
).
Default network activity, coupled with the frontoparietal control network, supports goal-directed cognition
.
Neuroimage
,
53
,
303
317
.
St-Laurent
,
M.
,
Adbi
,
H.
,
Bondad
,
A.
, &
Buchsbaum
,
B. R.
(
2014
).
Memory reactivation in healthy aging: Evidence of stimulus-specific dediffentiation
.
Journal of Neuroscience
,
34
,
4175
4186
.
St-Laurent
,
M.
,
Moscovitch
,
M.
,
Levine
,
B.
, &
McAndrews
,
M. P.
(
2009
).
Determinants of autobiographical memory in patients with unilateral temporal lobe epilepsy or excisions
.
Neuropsychologia
,
47
,
2211
2221
.
Staresina
,
B. P.
,
Henson
,
R. N.
,
Kriegeskorte
,
N.
, &
Alink
,
A.
(
2012
).
Episodic reinstatement in the medial temporal lobe
.
Journal of Neuroscience
,
32
,
18150
18156
.
Suddendorf
,
T.
, &
Corballis
,
M. C.
(
1997
).
Mental time travel and the evolution of the human mind
.
Genetic, Social, and General Psychology Monographs
,
123
,
133
167
.
Svoboda
,
E.
,
McKinnon
,
M. C.
, &
Levine
,
B.
(
2006
).
The functional neuroanatomy of autobiographical memory: A meta-analysis
.
Neuropsychologia
,
44
,
2189
2208
.
Tanji
,
J.
, &
Shima
,
K.
(
1994
).
Role for supplementary motor area cells in planning several movements ahead
.
Nature
,
371
,
413
416
.
Teyler
,
T. J.
, &
DiScenna
,
P.
(
1986
).
The hippocampal memory indexing theory
.
Behavioral Neuroscience
,
100
,
147
154
.
Teyler
,
T. J.
, &
Rudy
,
J. W.
(
2007
).
The hippocampal indexing theory and episodic memory: Updating the index
.
Hippocampus
,
17
,
1158
1169
.
Tong
,
F.
, &
Pratte
,
M. S.
(
2012
).
Decoding patterns of human brain activity
.
Annual Review of Psychology
,
63
,
483
509
.
Tulving
,
E.
(
1984
).
Elements of episodic memory
.
New York
:
Oxford University Press
.
Tulving
,
E.
(
1985
).
Memory and consciousness
.
Canadian Psychology
,
26
,
1
12
.
Tulving
,
E.
(
2002
).
Episodic memory: From mind to brain
.
Annual Review of Psychology
,
53
,
1
25
.
Viard
,
A.
,
Piolino
,
P.
,
Desgranges
,
B.
,
Chetelat
,
G.
,
Lebreton
,
K.
,
Landeau
,
B.
, et al
(
2007
).
Hippocampal activation for autobiographical memories over the entire lifetime in healthy aged subjects: An fMRI study
.
Cerebral Cortex
,
17
,
2453
2467
.
Vilberg
,
K. L.
, &
Rugg
,
M. D.
(
2007
).
Dissociation of the neural correlates of recognition memory according to familiarity, recollection, and amount of recollected information
.
Neuropsychologia
,
45
,
2216
2225
.
Vilberg
,
K. L.
, &
Rugg
,
M. D.
(
2008
).
Memory retrieval and the parietal cortex: A review of evidence from a dual-process perspective
.
Neuropsychologia
,
46
,
1787
1799
.
Vilberg
,
K. L.
, &
Rugg
,
M. D.
(
2009
).
Left parietal cortex is modulated by amount of recollected verbal information
.
NeuroReport
,
20
,
1295
1299
.
Vilberg
,
K. L.
, &
Rugg
,
M. D.
(
2012
).
The neural correlates of recollection: Transient versus sustained fMRI effects
.
Journal of Neuroscience
,
32
,
15679
15687
.
Wagner
,
A. D.
,
Shannon
,
B. J.
,
Kahn
,
I.
, &
Buckner
,
R. L.
(
2005
).
Parietal lobe contributions to episodic memory retrieval
.
Trends in Cognitive Sciences
,
9
,
445
453
.
Wheeler
,
M. A.
,
Stuss
,
D. T.
, &
Tulving
,
E.
(
1997
).
Toward a theory of episodic memory: The frontal lobes and autonoetic consciousness
.
Psychological Bulletin
,
121
,
331
354
.
Wing
,
E. A.
,
Ritchey
,
M.
, &
Cabeza
,
R.
(
2015
).
Reinstatement of individual past events revealed by the similarity of distributed activation patterns during encoding and retrieval
.
Journal of Cognitive Neuroscience
,
27
,
679
691
.
Xing
,
Y.
,
Ledgeway
,
T.
,
McGraw
,
P. V.
, &
Schluppeck
,
D.
(
2013
).
Decoding working memory of stimulus contrast in early visual cortex
.
Journal of Neuroscience
,
33
,
10301
10311
.
Xue
,
G.
,
Dong
,
Q.
,
Chen
,
C.
,
Lu
,
Z. L.
,
Mumford
,
J. A.
, &
Poldrack
,
R. A.
(
2013
).
Complementary role of frontoparietal activity and cortical pattern similarity in successful episodic memory encoding
.
Cerebral Cortex
,
23
,
1562
1571
.
Yin
,
H. H.
(
2010
).
The sensorimotor striatum is necessary for serial order learning
.
The Journal of Neuroscience
,
30
,
14719
14723
.
Yonelinas
,
A. P.
(
2002
).
The nature of recollection and familiarity: A review of 30 years of research
.
Journal of Memory and Language
,
46
,
441
517
.
Yonelinas
,
A. P.
,
Aly
,
M.
,
Wang
,
W. C.
, &
Koen
,
J. D.
(
2010
).
Recollection and familiarity: Examining controversial assumptions and new directions
.
Hippocampus
,
20
,
1178
1194
.