Abstract

When we have a rich and vivid memory for a past experience, it often feels like we are transported back in time to witness once again this event. Indeed, a perfect memory would exactly mimic the experiential quality of direct sensory perception. We used fMRI and multivoxel pattern analysis to map and quantify the similarity between patterns of activation evoked by direct perception of a diverse set of short video clips and the vivid remembering, with closed eyes, of these clips. We found that the patterns of distributed brain activation during vivid memory mimicked the patterns evoked during sensory perception. Using whole-brain patterns of activation evoked by perception of the videos, we were able to accurately classify brain patterns that were elicited when participants tried to vividly recall those same videos. A discriminant analysis of the activation patterns associated with each video revealed a high degree (explaining over 80% of the variance) of shared representational similarity between perception and memory. These results show that complex, multifeatured memory involves a partial reinstatement of the whole pattern of brain activity that is evoked during initial perception of the stimulus.

INTRODUCTION

When a memory for a prior experience is good, vivid, and rich in sensory details, we often say that it feels as if we are revisiting the past, that we are re-living the moment. As Vladimir Nabokov wrote in Lolita, a memory for a beloved face can be so precise, so evocative that it feels like an “optical replica” projected on the “dark innerside of your eyelids.” Indeed, one might say that memory is reliable and effective only to the extent that it mimics and emulates the conscious experience of which it is an imperfect copy (Place, 1956). The apparent similarity, at the phenomenological level, between direct perception and vivid memory seems to imply a fundamental correspondence between these two states at the neurological level (Ruchkin, Grafman, Cameron, & Berndt, 2003; Kosslyn, Ganis, & Thompson, 2001; Damasio, 1989; Hebb, 1968). However, we rarely confuse the feeling that we are recalling a past event with the feeling that we are directly experiencing the outside world (Tulving, 2002), a fact that implies that perception and memory are not identical at the neural level.

Previous work indicates that certain areas of the brain—which are known to be metabolically active during the perception of a particular stimulus category (e.g., faces), feature, or modality—are then “reactivated” when this stimulus category, feature, or modality is remembered (Ranganath, Cohen, Dam, & D'Esposito, 2004; Slotnick, 2004; Ishai, Ungerleider, & Haxby, 2000; Nyberg, Habib, McIntosh, & Tulving, 2000; Wheeler, Petersen, & Buckner, 2000). And this phenomenon appears to hold across both short (i.e., a few seconds; Buchsbaum, Olsen, Koch, & Berman, 2005; Rama & Courtney, 2005; Druzgal & D'Esposito, 2003) and longer timescales (i.e., at least a day; Wheeler et al., 2000). Other evidence, from neuropsychology and functional neuroimaging, however, suggests, on the contrary, that the brain regions that underlie conscious memory are at least partially distinct from those that support direct perception (Schumacher et al., 1996; Baddeley & Wilson, 1993; Warrington & Shallice, 1969). For instance, some patients can perceive and comprehend auditory speech and yet have severely reduced memory spans, a finding that argues for the possibility of a separate neurophysiological substrate for perception and memory storage processes (Shallice & Vallar, 1990; Warrington & Shallice, 1969). Nevertheless, the prevailing view in cognitive neuroscience is that at least a portion of the cortical machinery that is devoted to on-line stimulus analysis and perception is also utilized for the maintenance, retrieval, and storage of memory representations (Ruchkin et al., 2003; Kosslyn et al., 2001).

Real-life perceptual experiences—and our memories of these experiences—are complex, dynamic, multimodal, multifeatured, and, above all, idiosyncratic. To examine and precisely quantify the extent to which perception and memory share the same neural machinery requires an experimental strategy that captures the two corresponding states in their representational totality, allowing for the possibility that complex and variegated perceptual episodes—and the memories that reprise these episodes—emerge from the coordinated activity of distributed brain networks (Postle, 2006; Craik, 2002). In this study, we used fMRI and multivoxel pattern analysis (Rissman & Wagner, 2012; Chadwick, Hassabis, Weiskopf, & Maguire, 2010; O'Toole et al., 2007; Polyn, Natu, Cohen, & Norman, 2005; Hasson, Nir, Levy, Fuhrmann, & Malach, 2004; Strother et al., 2002; Haxby et al., 2001) to examine to what extent rich multimodal memories, when characterized in their totality as distributed patterns of neural activation, are neurophysiological simulacra of their perceptual counterparts.

We may posit two endpoints that form the boundaries of a hypothetical continuum describing the degree of congruence between brain activation at the time of perception and memory, respectively. On one end of this continuum lies the “shared machinery hypothesis,” which predicts that patterns of activation elicited during perception share a high degree of global similarity with the patterns of activation elicited when remembering these percepts. On the other extreme, the “separate machinery hypothesis” dictates that patterns of brain activity elicited by perception are highly dissimilar to activation patterns evoked during memory—a configuration indicating that the representational states are supported by different functional–anatomical systems and brain states. In line with previous research on the topic of cortical reactivation (for a review, see Danker & Anderson, 2010), we predict that memory involves a partial reinstatement of the patterns elicited by perception and that the brain regions participating in this mnemonic revisitation of perception are largely restricted to sensory and motor association regions as well as higher-order multimodal cortices (Meyer & Damasio, 2009; Damasio, 1989; Hebb, 1968). Thus, the brain patterns of activation elicited by perception and memory will be similar, but this similarity will be restricted to a subset of regions in the cerebral cortex. It is nevertheless still possible that the underlying abstract similarity structure, when considered as a lower-dimensional embedding of the cortical patterns evoked by perception and memory, respectively, will reveal a high degree of similarity between the two brain states (Kriegeskorte, Mur, & Bandettini, 2008) than is reflected by a literal pixel-by-pixel comparison of perception and memory.

The aim of the current study is to evaluate the similarity between perception and memory in a set of participants who were highly practiced at “mentally replaying” a set of relatively and multifeatural complex videos. We decided to extensively train participants to mentally replay the video stimuli because we wanted to maximize the phenomenological correspondence between a person's perception and memory. We reasoned that repeatedly viewing the set of videos would strengthen and improve the fidelity and chronological accuracy of participants' memories for the videos. In essence, then, the goal of this study was not to investigate a particular “type” of memory (e.g., episodic, autobiographical, semantic, imagery, working memory) but rather to quantify regional variation in the neurophysiological correspondence between perception and memory in the perhaps unusual condition in which the ability to “mentally replay” a complex memory is near the limit of human performance.

METHODS

Overview

The experiment consisted of three phases. In the first phase, participants (n = 20, eight women) were scanned while they watched a set of 12 video clips, each 9-sec long and each repeated 27 times. These clips were chosen for their diversity of content, which variously included videos with music, speech, faces, human emotion, body parts, biological motion, outdoor scenery, first person navigation, environmental sounds, eating, dancing, hand movements, plant life, animals walking, swimming, etc. (Table 1). A subset (n = 9; four women) of these 20 participants then underwent an intensive and structured memory-training regimen (Figure 1B) consisting of at least sixteen 1-hr sessions completed over the course of 2–3 months in which they practiced “mentally replaying” each of the videos that they watched during the initial fMRI scan. During memory training, participants also learned to associate each video with a unique symbolic cue (see Figure 1B).

Table 1. 

A Representation of the Diversity of the Content Matter of the Video Clips on Seven Stimulus Dimensions

Video
Speech
Music
Environmental Sounds
Biological Motion
Egocentric Motion
Face
Outdoor Scene
Baboons    ✓  a ✓ 
Ballet  ✓  ✓  ✓  
Basketball    ✓  ✓  
Brando ✓ ✓  ✓  ✓  
Car chase  ✓ ✓  ✓ ✓ ✓ 
Cliff     ✓  ✓ 
Comedian ✓  ✓ ✓  ✓  
Flatworms 
McEnroe ✓  ✓ ✓  ✓ ✓ 
Model    ✓  ✓ ✓ 
Origami    ✓    
Rose 
Video
Speech
Music
Environmental Sounds
Biological Motion
Egocentric Motion
Face
Outdoor Scene
Baboons    ✓  a ✓ 
Ballet  ✓  ✓  ✓  
Basketball    ✓  ✓  
Brando ✓ ✓  ✓  ✓  
Car chase  ✓ ✓  ✓ ✓ ✓ 
Cliff     ✓  ✓ 
Comedian ✓  ✓ ✓  ✓  
Flatworms 
McEnroe ✓  ✓ ✓  ✓ ✓ 
Model    ✓  ✓ ✓ 
Origami    ✓    
Rose 

aBaboon faces.

Figure 1. 

Detailed overview of study design. The study was conducted in three parts. (A, left) First, 20 participants were scanned while passively viewing the 12 videos which were repeatedly presented in pseudorandom order. (A, right) After memory training, participants were scanned an additional three times. During each of these scans, a mental replay task (eight runs per day; a total of 24) and the passive viewing task (two runs per day; a total of 6) were done. The mental replay task consisted of the presentation of the cue, followed by a 9-sec mental replay period and a memory rating. (B) Training involved the repeated performance of a task that was divided into three phases. In Phase 1, participants viewed a symbolic cue followed by its associated video. In Phase 2, participants practiced, mentally replaying each video. In Phase 3, participants were tested on their ability to mentally replay the videos in real time. This was achieved by asking participants to mentally replay the cued video and then interrupting the replay (the word “PAUSE” displayed on the screen) at some point between 300 and 8700 msec later. Two images were shown side by side, and participants had to choose which image was closer in time to the state of their mental replay at the time of the interruption.

Figure 1. 

Detailed overview of study design. The study was conducted in three parts. (A, left) First, 20 participants were scanned while passively viewing the 12 videos which were repeatedly presented in pseudorandom order. (A, right) After memory training, participants were scanned an additional three times. During each of these scans, a mental replay task (eight runs per day; a total of 24) and the passive viewing task (two runs per day; a total of 6) were done. The mental replay task consisted of the presentation of the cue, followed by a 9-sec mental replay period and a memory rating. (B) Training involved the repeated performance of a task that was divided into three phases. In Phase 1, participants viewed a symbolic cue followed by its associated video. In Phase 2, participants practiced, mentally replaying each video. In Phase 3, participants were tested on their ability to mentally replay the videos in real time. This was achieved by asking participants to mentally replay the cued video and then interrupting the replay (the word “PAUSE” displayed on the screen) at some point between 300 and 8700 msec later. Two images were shown side by side, and participants had to choose which image was closer in time to the state of their mental replay at the time of the interruption.

Following this extensive memory training, participants were brought back for three additional 1-hr fMRI sessions (carried out over the span of 1 week) in which they “mentally replayed” each of the videos. In this task, participants were first cued with the learned associates and asked to close their eyes and mentally replay—in as vivid and temporally accurate a manner as possible—the video clip to which the symbol referred. Following each mental replay attempt, participants pressed a button indicating on a scale of 1–4 (“1” = poor memory and “4” = excellent memory) the vividness and accuracy of their attempt to replay the cued video clip (Chadwick, Hassabis, & Maguire, 2011; Hassabis, Kumaran, & Maguire, 2007). On each of these three 1-hr follow-up fMRI scans, participants performed the same cued memory replay task, yielding a total of 36 trials (accumulated across the three sessions) for each of the 12 video clips. After completing eight runs of the memory task for each of the three posttraining scans, participants then performed two additional runs (six total across the 3 days) in which they passively viewed the 12 video clips just as they had done in the first pretraining fMRI scan (Figure 1B). On each of the 3 days, the two passive viewing were always performed last, so that no mental replay trials followed a passive viewing trial within a scanning session. We added the six runs of “posttraining” perception scans to allow for the explicit comparison between activation patterns elicited by passive perception of the videos before and after the extensive training procedures.

Participants

Twenty (eight women, mean age = 22.8 years, range = 18–36 years) subjects participated in the first fMRI experiment and a subset of nine of these subjects (five men, mean age = 24.2 years, range = 18–36 years) participated in the memory training and the second fMRI experiment. No participant had any history of neurological or psychiatric disease. All participants were right-handed and had normal or corrected-to-normal vision. Informed consent was obtained from all participants, and the local ethics board at the Rotman Research Institute at Baycrest Hospital approved the study.

Video Stimuli

A set of 12 video clips were gathered from on-line sites (such as Vimeo.com and YouTube.com) and chosen so as to create a small set of videos with diverse content matter. Some of the major perceptual and semantic features of these videos are listed in Table 1. All videos were of good quality and were converted if necessary so as to have the same screen resolution (960 × 540 pixels) and format (Quicktime). During both memory training and the second fMRI experiment, 12 abstract symbols were used as associated cues for each of the 12 videos. These nonverbalizeable symbols were selected among the characters in the “Wingdings” font in Microsoft Word and chosen to be clearly distinct from one another.

fMRI Scanning: General

In all fMRI scans, visual stimuli were back-projected on to a screen behind the scanner and viewed by the participant through a mirror mounted on the head coil. The display had a spatial resolution of 1024 × 768 pixels and a visual angle of approximately 14.8° × 12.1° at a viewing distance of 132 cm. Audio stimuli were delivered from the PC running the experimental task through Avotec magnetic-safe headphones at a comfortable listening level.

MRI Data Acquisition

Imaging data were acquired on a 3.0-T Siemens MAGNETOM Trio MRI scanner at the Rotman Research Institute at Baycrest Hospital. T2*-weighted EPIs sensitive to BOLD contrast were acquired using a 32-channel transverse electromagnetic send-and-receive radiofrequency head coil system. Images were acquired using a two-shot gradient-echo EPI sequence (22.5 × 22.5 cm field of view with a 96 × 96 matrix size, resulting in an in-place resolution of 2.5 × 2.5 mm for each of twenty-four 3.5-mm axial slices with a 1-mm interslice gap; repetition time = 1.5 sec; echo time = 27 msec; flip angle = 62°). High-resolution gradient-echo multislice T1-weighted scans, coplanar with the EPIs, as well as whole-brain magnetization prepared rapid gradient echo (MP-RAGE) 3-D T1-weighted scans were acquired for anatomical localization.

fMRI Scan 1: Passive Perception

In the first fMRI experiment (Figure 1A), participants were told that they would be shown a number of videos, some of which would be accompanied by sound, and that they were to play close attention to each of the videos. Participants were also informed that there would be a test on the content matter of the videos at the conclusion of the scan. Before the start of the scan, participants were shown each of the 12 videos twice to familiarize them with the stimuli. During the scan, participants viewed (and listened, for those videos that had sound) to each of the 12 videos a total of 27 times over nine 7-min fMRI scanning runs. Each video was presented three times per scanning run in pseudorandom order. Between each video, a white crosshair was presented on the center of the screen, and participants instructed to fixate on it for the duration of a variable (1–4 sec ISI). After the scan, a test consisting of several short-answer questions for each video was administered to the participant. The purpose of this test was simply to establish that participants were attending to the videos.

fMRI Scan 2: Mental Replay

Following completion of memory training (see below), a subset (n = 9) of the original group of participants were scheduled for three separate fMRI sessions (all conducted within the span of 2 weeks) in which they performed a task requiring “mental replay” of the 12 videos. On each trial in the mental replay task (Figure 1A, right), participants were first shown 1 of the 12 symbols that they had learned to associate with the videos during the training phase. This was followed by a 10-sec period in which participants were instructed to first close their eyes and then to mentally replay the cued video with maximum attention to the visuospatial and temporal characteristics of the original stimulus. Participants were instructed to open their eyes and press the left button when their mental replay had concluded or when they heard a tone, whichever came first. A 500-msec (800 Hz) tone always followed the offset of the symbol by 10 sec or exactly 1 sec longer than the true duration of the video. Thus, if the mental replay exceeded the duration of the video by more than 1 sec, the tone indicated for participants to open their eyes and fixate on the center of the screen. After the tone, a fixation cross appeared on the screen for 2 sec, followed by the words “Rate Your Memory.” Participants then provided a rating of the quality of their memory for the video on a scale of 1–4: 1 = poor, 2 = OK, 3 = good, 4 = excellent. Participants were instructed to base their ratings in terms of the relative quality of their memory compared with what they thought to be the best of their capability, as had been established during the memory training sessions. After completing the eight runs of the mental replay task participants performed two additional 7-min runs of the passive perception task for the purpose of comparing with the pretraining fMRI data. At the end of the three sessions, participants performed 36 trials (across twenty-four 7-min runs) for each of the 12 symbols and an additional 18 trials (across six 7-min runs) of passive viewing.

Memory Training

After completion of the first fMRI scan, 11 subjects were recruited to participate in an extended memory training regimen followed by three posttraining fMRI scans; 9 of these 11 recruited participants completed the minimum required number of training sessions (18 sessions). Each training session took approximately 1 hr to complete and was designed to achieve the following goals: (1) for participants to associate an arbitrary symbol with each of the 12 videos, (2) provide an opportunity for extensive practice in recalling and “mentally replaying” each of the 12 videos, and (3) to provide an objective assessment of the accuracy of the participants' memory for the videos. To achieve these aims, the training task was divided into three phases (Figure 1B). During Phase 1, the symbol associated with each video was presented and then followed by the 9-sec video itself. Participants were instructed to note the association between the symbol and video and then to simply watch and study to the video itself. During Phase 2, the symbol associated with the video was presented, followed by the video itself, followed by the symbol again, and then by a cue (a green crosshair) to begin “mentally replaying” with their eyes closed the just-presented video. When participants were finished replaying (e.g., when the participant's mental replay of the video “ended”), participants were instructed to press the “1” key. After this, the symbol was shown once again and the participants mentally replayed the video once again. This sequence of events (symbol, video, symbol, Mental Replay 1, symbol, Mental Replay 2) was repeated twice for each of the 12 videos.

In the final phase, participants were first shown in a pseudorandom order a video's associated symbol, then the video itself, followed by a cue to start “mentally replaying” the video with eyes open and fixated at the center of the screen. After a variable duration between 300 msec and 8.7 sec, the word “PAUSE” would appear on the center of the screen followed by two still frames taken from the current video. The participant then had to choose which of the still frames was closer in time to the current state of their “mental replay.” Thus, if participants were able to “replay” the videos with good temporal precision, they should be able to reliably select the still frame that was nearer (in actual elapsed time) to their own internal representation of the video. The purpose of this test was to track the temporal fidelity with which participants could mentally replay each of the videos. The difficulty of this decision depended on the difference in actual time between the two still frames. Thus, if the frames were far apart in time (i.e., greater than 4 sec) it was relatively easy to select the frame that was closer to the current state of one's mental replay. The distribution of the difference in time between the two frames was exponential so that that there were many more trials where the difference was relatively small (i.e., <1 sec) than trials for which the difference was large (i.e., >4 sec).

Participants were asked to perform three training sessions per week over the course of 6 weeks for a total of 18 sessions. The training task was installed on participants' laptop computers so that they could perform the training regimen at their convenience at home. Participants were required to send the log file containing behavioral data that was generated for each training session via e-mail to an address that was checked regularly by the experimenters. In this way, compliance on performing the training sessions was monitored.

It was not always possible to immediately schedule a participant for the posttraining fMRI scan on completion of training, and in these cases, participants continued to take part in the training sessions (twice a week) until they could be scheduled. Thus, the total number of training sessions varied somewhat from participant to participant (mean = 22.5, min = 18, max = 30). At the end of training, participants were given a test consisting of questions of varying difficulty regarding the content matter of the videos. The purpose of this test was to verify that participants had formed detailed memories of each of the clips and were capable of retrieving sensory details about each of the videos. In addition, to verify that participants had learned the relation between each symbolic cue and its associated video, participants were given a test requiring them to match each of the symbols with a verbal descriptor of each of the videos. All participants performed this symbol–video matching task perfectly.

Image Processing and Statistical Analysis

For each scanning run, 302 EPI images were obtained in DICOM format and exported using AFNI (Cox, 1996) into NIFTI-1 format. For each participant, images were motion-corrected and realigned to the first image of the first run of the session using the AFNI program 3dvolreg. Functional images were then smoothed with a 5-mm FWHM Gaussian kernel. All statistical analyses were performed on these smoothed and realigned images.

Each participant's high-resolution anatomical MP-RAGE scan was normalized to the Montreal Neurological Institute (MNI) stereotaxic space with a nonlinear transformation using nonlinear symmetric normalization as implemented in ANTS (Avants, Epstein, Grossman, & Gee, 2008). An additional six-parameter rigid-body registration between each participant's mean functional image, derived from the first functional run, was performed to derive a transformation between each participant's native EPI space and the participant's high-resolution anatomical image. For display purposes, data from group analyses were projected on to a standard MNI surface template using the AFNI program 3dVol2Surf and visualized using the AFNI program SUMA.

All participants' structural MP-RAGE images were segmented in to gray, white, and cerebrospinal fluid using FMRIB software library (version 4.0, www.fmrib.ox.ac.uk/fsl) using the program “fast.” The gray matter images were warped to MNI space and averaged to create a gray matter probability image for the full group (n = 20). This probability image was then thresholded at .2 to create a gray matter mask in normalized MNI space. This mask was used to restrict the voxel space to gray matter areas in multivariate analyses conducted at the group level.

Univariate Statistical Analyses

Single-subject multiple regression modeling was carried out using the AFNI program 3dDeconvolve. Each of the conditions (e.g., video perception, mental replay) was modeled by convolving a hemodynamic response function (SPM canonical function as implemented in AFNI) with the onset and duration of the experimental events. An additional set of nuisance regressors (a constant term plus linear, quadratic, and cubic polynomial terms) was included for each scanning run to model low-frequency noise in the time series data. Statistical contrasts at the single-subject level were computed as weighted sums of the estimated beta coefficients divided by an estimate of the standard error, yielding a t statistic for each voxel in the image volume. Random effects group analyses were then carried out on the spatially normalized single-subject contrast images using a one-sample t test (against 0) for simple hypotheses. Estimates for activity related either to perceiving or mentally replaying a video was computed as contrasts between each video and the average activity in all the other videos. Thus, for both the perception and memory data sets, 12 contrast images, one for each video was computed at both the single subject and group levels. In fMRI Experiment 1 (passive viewing), the events of interest were the set of videos, each with a 9-sec duration. In fMRI Experiment 2 (mental replay), two additional events were modeled but were not further analyzed: namely, the 500 msec tone indicating for participants to stop replaying and the memory rating response.

Univariate Conjunction Analysis of Perception and Memory Data Sets

To evaluate the tendency for regions active during perception to also be active during memory, a conjunction analysis was performed for every corresponding pair of contrasts in the video and memory data sets. For each video contrast at the single subject level, a voxel was labeled as conjointly active if the Z score for both perception and memory contrasts was greater than 1.65 (p < .05, one-tailed). At the group level, the number of participants with conjoint activation at each voxel was tallied and converted to a proportion. To assess statistical significance, the binomial distribution was used to assess the likelihood under the null hypothesis that there would be at least k participants showing conjoint activity at a given voxel. For the purpose of display (Figure 2), voxels are colored for which at least three of nine (33%) participants (p < .008) show conjoint activity.

Figure 2. 

(A) Cortical flat maps showing overlap between perception and memory for three of the videos. (B) Cortical flat map showing the voxelwise correlation between perception and memory. (A) For all three maps, green colors show regions activated during passive viewing of the videos and warm colors show regions are active during both passive viewing and mental replay of the videos. (B) The unthresholded cortical flat map shows for every voxel the correlation between activity at perception and memory across the 12 videos. Warm colors show regions with a positive correlation, and cool colors show negative correlations. A thresholded view of this map is shown in Figure 3 on an inflated cortical surface. CS = central sulcus; IFS = inferior frontal sulcus; STS = superior temporal sulcus; SF = sylvian fissure.

Figure 2. 

(A) Cortical flat maps showing overlap between perception and memory for three of the videos. (B) Cortical flat map showing the voxelwise correlation between perception and memory. (A) For all three maps, green colors show regions activated during passive viewing of the videos and warm colors show regions are active during both passive viewing and mental replay of the videos. (B) The unthresholded cortical flat map shows for every voxel the correlation between activity at perception and memory across the 12 videos. Warm colors show regions with a positive correlation, and cool colors show negative correlations. A thresholded view of this map is shown in Figure 3 on an inflated cortical surface. CS = central sulcus; IFS = inferior frontal sulcus; STS = superior temporal sulcus; SF = sylvian fissure.

Voxelwise Correlation Between Perception and Memory

To assess the extent to which the level of activity during perception (relative to the other videos) was related to the level of activity during memory, we computed for every participant a correlation between the 12 pairs of perception and memory contrasts (t statistics) at every voxel in the brain. A positive correlation indicates that there is a positive relationship in a given voxel between the level of activity during perception and the level of activity during memory. A negative correlation indicates that high activity during perception is associated with low activity during memory. The correlation maps were then Z-transformed and normalized to MNI space and submitted to a voxelwise t test at the group level. Statistical significance was assessed using the false discovery rate (Genovese, Lazar, & Nichols, 2002) of .05.

Pattern Reliability Analysis Using Whole-brain Correlations

To establish that particular videos, when viewed passively or when mentally replayed, gave rise to reliable distributed patterns of activation, a cross-validated “leave-one-out” correlation analysis was carried out separately for both the perception and memory data sets (Hastie, Tibshirani, & Friedman, 2001). These analyses were performed at the group level on gray matter voxels (using the gray matter mask described previously) in normalized MNI space and were performed in the same manner for the perception (n = 20) and memory (n = 9) data sets. For every participant, a matrix of correlations was computed between each participant's video contrast images (n = 12) and the average of the contrast images for the rest of the group. If the maximum correlation fell on the diagonal of the correlation matrix, the “left out” contrast for that participant was correctly classified. Classification accuracy for the group was defined as the number of correctly classified contrasts and was divided by the total number of contrasts (N × 12) to give a proportion.

Barycentric Discriminant Analysis of Multivariate Activation Patterns

Barycentric discriminant analysis (BADA) is a multivariate statistical method that is used to decompose a data matrix into orthogonal factors that capture the association between a set of J variables (voxels) and N categories (videos; Abdi, Williams, Gobbini, Dunlop, & Haxby, 2012; Abdi, 2010a). Each observation is represented by the 1 by J vector of the activation values of its voxels. Then each category of interest is represented by the barycenter of its observations (i.e., the weighted mean; the barycenter is also called the center of gravity of the observations of a given category). A generalized PCA is finally performed on this category by variable matrix. This analysis gives a set of discriminant factor scores for the categories and another set of factor scores for the voxels. The original observations are then projected onto the category factor space, providing a set of factor scores for the observations. The Euclidean distance of each observation to the set of categories is computed from the factor scores, and each observation is assigned to its closest category.

BADA: Single-subject Analyses

We applied BADA to each of the single-subject data sets for which perception and memory fMRI data existed (n = 9) to assess the similarity of the factor spaces between perception and memory data sets. Thus, for each participant, a BADA was separately applied to the 12 contrast maps for the perception and the 12 contrast maps for memory. The two 12 × 12 matrices representing the factor spaces for the two BADA solutions were then compared with each other using a metric of multivariate similarity called the RV coefficient (which can be interpreted as the matrix equivalent of a squared coefficient of correlation and a permutation test was used to assess its significance; Abdi, 2010b). In addition, we evaluated the ability for the BADA solution derived from the perception data to classify the contrast maps for the memory data (and vice versa). This was achieved by first projecting the memory data set into the discriminant factor space derived from the perception data set. Then we computed the Euclidean distance between the projected (memory) factor scores and the factor scores for the original solution (perception). The classification of a memory scan is accurate when the distance between the corresponding memory and the perception videos is smaller than the distance between any of the other (N − 1) videos.

BADA: Group Analysis

To derive a set of factor scores (for the videos) and associated factor loadings (for the voxels) that are common to the group of participants, we computed a BADA solution in normalized MNI space including all gray matter voxels. We computed separate solutions for the perception, memory, and posttraining perception data sets on each of the nine participants that completed memory training. To compare the similarity of the factor spaces derived from these solutions, we again used the RV coefficient. We also computed a “compromise” BADA solution for the perception, memory, and posttraining perception. In this compromise analysis, each data set was first subjected to a singular value decomposition and normalized by dividing each datum in this set by the first singular value its data set (see Abdi, Williams, Valentin, & Bennani-Dosse, 2012, for a justification of this normalization). Then the three data sets were concatenated into a single matrix by averaging them, and a BADA solution was applied to the full data set (Abdi et al., 2012). Finally, each of the data subsets was then projected into the factor space of the full solution, allowing for the assessment, visualization, and relative relationships of the three experimental data sets.

Regional Similarity Analysis of Perception and Memory

To identify brain regions that showed a high degree of similarity between perception and memory, we performed a local BADA analysis for every participant and every ROI in the Harvard–Oxford probabilistic atlas of 48 cortical and 6 subcortical areas in both hemispheres. For each ROI and participant, we computed the RV coefficient (and associated p value) reflecting the degree of similarity between the factor spaces for the perception and memory data. Statistical significance at the group level was determined for each ROI by tallying the number of participants with significant RV coefficients (p < .05), and this count was converted to a probability using the binomial distribution.

RESULTS

Analysis of Behavioral Training Data

Every behavioral training session concluded with a test of the temporal precision of mental replay (see Figure 1B). Participants were cued with one of the symbols denoting a video and at the offset of this cue began mentally replaying the video. After some variable interval between 300 and 8700 msec after cue offset, participants were instructed to “pause” their mental replay and were then presented with two still frames from the video, one of which was in sync with the true timing of the video and one that was not. Participants had to choose which frame was in sync with their mental replay of the video. The difference in time (ΔT) separating the two frames is a measure of the difficulty of the choice. If the participants' ability to mentally replay the videos improved with practice, we would expect an improvement in choosing the correct frame, especially when the time difference is small. To test for this effect, we conducted a logistic linear mixed model analysis, with participant as a random effect, and the log of the training session number and ΔT as continuous variables. The response variable was 1 if the trial was correct and 0 if it was incorrect. There was a main effect of ΔT (Z = 5.91, p < .0001), indicating that the larger the temporal separation between the two still frames, the better the accuracy. There was no main effect of session, but there was a ΔT by session interaction (Z = 3.52, p < .0004). To localize where along the range of time differences learning occurred, we binned the ΔT variable into six quantiles and computed the logistic regression model for each quantile, using only log session as an independent variable. This analysis showed that significant learning occurred only for the largest two quantiles of ΔT, which included values of ΔT larger than 1.34 sec (see Table 2). Thus, the temporal precision of mental replay increased only for trials in which the difference in timing between the two still frames was greater than 1.34 sec.

Table 2. 

Results of Logistic Regression Analysis for Test Phase of Memory Training Data

dT Quantile
Accuracy
z Score
p
0.324–0.334 .49 −0.85 .4 
0.334– 0.343 .53 −0.47 .64 
0.343–0.676 .54 1.08 .28 
0.676–1.34 .62 1.00 .32 
1.34–2.67 .68 2.76 .0058 
2.67–8.33 .83 3.33 .0008 
dT Quantile
Accuracy
z Score
p
0.324–0.334 .49 −0.85 .4 
0.334– 0.343 .53 −0.47 .64 
0.343–0.676 .54 1.08 .28 
0.676–1.34 .62 1.00 .32 
1.34–2.67 .68 2.76 .0058 
2.67–8.33 .83 3.33 .0008 

“dT quantile” refers to the interval differences (dT) between the two test frames and corresponds to one sixth of the data set; “accuracy” is the proportion correct for the quantile; “z score” is the test statistic for the effect of training session number on accuracy for the dT quantile; p value is the significance of the effect of training session number on accuracy for the dT quantile. The range of the dT quantiles increases by row due the fact that dT had an exponential distribution.

Analysis of Behavioral Ratings during fMRI Scan 2 (Mental Replay)

Following each trial in the fMRI mental replay task, participants provided a rating on a scale of 1–4 indicating how well they had succeeded in recalling the video relative to how well they were capable of recalling that video. For every participant, we computed the mean rating for each video and then carried out a one-way ANOVA, with mean rating as the dependent variable and video as the independent variable, to test for differences in ratings across videos. The main effect of video was significant, F(11, 88) = 2.88, p = .0028. Mean ratings and standard deviations are reported in Table 3.

Table 3. 

Means and Standard Deviations for Subject Memory Ratings from fMRI Mental Replay Experiment

Video
Mean Rating
SD
Rose 2.70 0.58 
Baboons 2.74 0.44 
Car chase 2.85 0.43 
Ballet 2.88 0.51 
Flatworms 2.907 0.40 
Basketball 2.94 0.62 
Cliff 2.96 0.47 
Brando 3.00 0.40 
Model 3.02 0.46 
Mcenroe 3.06 0.5 
Origami 3.18 0.47 
Comedian 3.22 0.4 
Video
Mean Rating
SD
Rose 2.70 0.58 
Baboons 2.74 0.44 
Car chase 2.85 0.43 
Ballet 2.88 0.51 
Flatworms 2.907 0.40 
Basketball 2.94 0.62 
Cliff 2.96 0.47 
Brando 3.00 0.40 
Model 3.02 0.46 
Mcenroe 3.06 0.5 
Origami 3.18 0.47 
Comedian 3.22 0.4 

Reliability of Whole-brain Activation Patterns

Our first goal was to show that patterns of activity associated with perceiving and mentally replaying the videos, when examined independently of each other, are both reliable across participants and can be used to predict which video clips participants were perceiving or remembering (Strother et al., 2002). Starting with the perception data, we calculated whole-brain univariate activity for the set of 20 participants scanned while passively viewing the videos. Activation images were expressed as the average activity for each video subtracted by the average activity across all videos. These video-specific activation maps were then combined across participants using a one-sample t test. To demonstrate that these spatial activation patterns were both reliable across participants and could be used to classify activation patterns in an independent sample, a leave-one-out cross-validation approach was used (Hastie et al., 2001). For each video-specific activation map, we computed a Pearson correlation coefficient between 12 group averaged activation patterns for N − 1 participants to the 12 activation patterns for the left-out participant, yielding a set of similarity measures for all participants and all videos. The average correlation across all videos and participants was .65, and our ability to classify the activation maps of the independent samples was 96.7% accurate. This same analysis was then repeated for the memory data, resulting in an average leave-one-out correlation of .3 and an average classification accuracy of 71%. Thus, the activation patterns for both perception and memory of the videos were reliable across participants and could be used to discriminate between brain patterns associated with passive viewing or mental replay of the different videos, respectively.

Global Pattern Similarity between Perception and Memory

Having established that both perception and memory of the videos gave rise to reliable patterns of activation when considered separately, we sought to confirm the hypothesis that patterns of activation during perception and memory of the videos were globally similar to one another. A simple quantifiable formulation of the general hypothesis that brain regions active during perception are reactivated during memory is the conditional probability P(M|P) that a voxel is active during memory given that it is active during perception (normalized by the unconditional probability that it is active during memory):
formula
We calculated P(M|P) for all 12 pairs of perception and memory activation maps, resulting in an average odds ratio of 3.92 (SD = 1.01), which was statistically significant (log odds = 1.33, t = 15.06, p < .0001). The inverse conditional probability (i.e., the probability that a perception voxel is active, given that it is active during memory) showed an even stronger association, yielding an average odds ratio of 5.86 (log odds = 1.71, t = 15.20, p < .0001). Thus, knowing that a voxel is active during memory is a better indication that it was also active during perception than the reverse.

Voxel by Voxel Correlations between Perception and Memory

To identify more precisely the specific brain regions that are reactivated during memory, we first computed a voxel-wise conjunction (Nichols, Brett, Andersson, Wager, & Poline, 2005) between perception and memory activation maps for each of the 12 videos (Figure 2) where it can be seen that many areas that activate during perception also activate during memory. To assess the relationship between activity at perception with activity during memory, we computed the voxel-wise Pearson correlation between all 12 perception maps and all 12 memory maps for every participant. To test for group significance, a one-sample t test was computed on the set of r-to-Z transformed correlation coefficients for every voxel in MNI space. The resultant group perception/memory correlation map (Figure 2B, unthresholded; Figure 3, thresholded) offers a picture of the distributed cortical topography of memory reactivation and reveals that areas of reactivation tend to lie in regions of unimodal and multimodal association cortex, whereas primary auditory and visual cortices actually show an inverse correlation between perception and memory activity levels. A complete listing of significantly active regions (false discovery rate, q = .05 and cluster size > 10 voxels) is in Table 4.

Figure 3. 

Surface view of voxelwise (false discovery rate, q < .05) correlations between perception and memory contrasts. Left and right hemisphere views of the cortical surface showing regions of significant voxelwise correlations between activation at perception and memory. Warm colors show positive correlations; cold colors show negative correlations. See Table 4 for all peak coordinate locations.

Figure 3. 

Surface view of voxelwise (false discovery rate, q < .05) correlations between perception and memory contrasts. Left and right hemisphere views of the cortical surface showing regions of significant voxelwise correlations between activation at perception and memory. Warm colors show positive correlations; cold colors show negative correlations. See Table 4 for all peak coordinate locations.

Table 4. 

Peak Coordinates in Clusters Voxelwise Correlation between Perception and Memory Activation Maps


X
Y
Z
Z score
Region 
Precuneous cortex 20.5 −59.5 21.5 6.75 
Sup. lateral occipital cortex −33.5 −83.5 27.5 6.29 
Post. supramarginal gyrus −51.5 −41.5 21.5 5.89 
Post. middle temporal gyrus −51.5 −41.5 3.5 5.74 
Precentral gyrus −27.5 −11.5 48.5 5.42 
Post. Parahippocampal gyrus 29.5 −35.5 −14.5 5.26 
Temporal pole −54.5 12.5 −8.5 5.17 
Superior frontal gyrus 23.5 −2.5 57.5 4.93 
Frontal pole 53.5 39.5 9.5 4.44 
Post. temporal fusiform cortex −33.5 −41.5 −8.5 4.25 
Inferior frontal gyrus, pars opercularis −54.5 12.5 27.5 4.16 
Paracingulate gyrus −12.5 45.5 0.5 4.07 
Temporal occipital fusiform cortex 44.5 −47.5 −26.5 3.99 
Post. cingulate gyrus −3.5 −20.5 45.5 3.91 
Globus pallidus 14.5 −20.5 12.5 3.90 
Postcentral gyrus 53.5 −11.5 39.5 3.82 
Frontal medial cortex 11.5 54.5 −8.5 3.77 
Ant. cingulate gyrus −3.5 42.5 12.5 3.72 
Inferior frontal gyrus, pars triangularis −51.5 30.5 0.5 3.71 
Inferior temporal gyrus, temporooccipital 47.5 −50.5 −8.5 3.52 
Frontal operculum cortex −45.5 27.5 0.5 3.47 
Inf. lateral occipital cortex −51.5 −77.5 15.5 3.34 
Ant. temporal fusiform cortex 32.5 3.5 −41.5 3.27 
Middle frontal gyrus 35.5 0.5 54.5 3.10 
 
Negative Correlations 
Occipital pole 20.5 −92.5 6.5 −3.62 
Planum temporale −54.5 −23.5 9.5 −3.52 
Heschl's gyrus −39.5 −23.5 3.5 −3.50 

X
Y
Z
Z score
Region 
Precuneous cortex 20.5 −59.5 21.5 6.75 
Sup. lateral occipital cortex −33.5 −83.5 27.5 6.29 
Post. supramarginal gyrus −51.5 −41.5 21.5 5.89 
Post. middle temporal gyrus −51.5 −41.5 3.5 5.74 
Precentral gyrus −27.5 −11.5 48.5 5.42 
Post. Parahippocampal gyrus 29.5 −35.5 −14.5 5.26 
Temporal pole −54.5 12.5 −8.5 5.17 
Superior frontal gyrus 23.5 −2.5 57.5 4.93 
Frontal pole 53.5 39.5 9.5 4.44 
Post. temporal fusiform cortex −33.5 −41.5 −8.5 4.25 
Inferior frontal gyrus, pars opercularis −54.5 12.5 27.5 4.16 
Paracingulate gyrus −12.5 45.5 0.5 4.07 
Temporal occipital fusiform cortex 44.5 −47.5 −26.5 3.99 
Post. cingulate gyrus −3.5 −20.5 45.5 3.91 
Globus pallidus 14.5 −20.5 12.5 3.90 
Postcentral gyrus 53.5 −11.5 39.5 3.82 
Frontal medial cortex 11.5 54.5 −8.5 3.77 
Ant. cingulate gyrus −3.5 42.5 12.5 3.72 
Inferior frontal gyrus, pars triangularis −51.5 30.5 0.5 3.71 
Inferior temporal gyrus, temporooccipital 47.5 −50.5 −8.5 3.52 
Frontal operculum cortex −45.5 27.5 0.5 3.47 
Inf. lateral occipital cortex −51.5 −77.5 15.5 3.34 
Ant. temporal fusiform cortex 32.5 3.5 −41.5 3.27 
Middle frontal gyrus 35.5 0.5 54.5 3.10 
 
Negative Correlations 
Occipital pole 20.5 −92.5 6.5 −3.62 
Planum temporale −54.5 −23.5 9.5 −3.52 
Heschl's gyrus −39.5 −23.5 3.5 −3.50 

Ant. = anterior; Inf. = inferior; Post. = posterior; Sup. = superior.

BADA of Perception and Memory Patterns

Although the foregoing univariate analyses offer strong evidence for distributed cortical reactivation during memory for complex perception events, we turned to multivariate pattern analysis methods to characterize the multidimensional space in which perception and memory are embedded. Using BADA, we computed separate solutions for the perception and memory data sets for each participant. We directly compared the factor spaces for these participant-specific BADA solutions for perception and memory data sets using the RV coefficient. Brain activation patterns for perception shared 91% of the variance (mean RV = .91, SD = .026; all p < .01) with activation patterns for memory. Using the BADA solution derived only from the perception data to classify the memory activation patterns resulted in 35% classification accuracy (chance = 8.3%); and using the memory solution to classify the perceptual activation patterns was 38% accurate. If reactivation of perceptual brain patterns during memory is only partial, however, then classification performance should improve if the model is trained on a subset of voxels that are active during the memory task. Here we define “active” as the voxels that show significant differential activity across videos. A simple measure of this differential activity is the standard deviation of the normalized activation level (t statistics) for the set of video contrasts. When the BADA solution was trained using only the top 5% active voxels for the memory data set, classification accuracy rose to 58% (memory solution to classify perception) and 59% (perception solution to classify memory), respectively. To evaluate the full relationship between the percentage threshold applied to the memory data set and classification accuracy, we computed the BADA solution for cutoff values ranging from 0% (include all voxels in the brain mask) to 99% (include only the top 1% of active memory voxels; see Figure 4).

Figure 4. 

Classification of perceptual brain patterns using memory patterns. The plot shows the ability, in terms of classification accuracy, for discriminant solutions trained on memory to classify activation maps derived from the perception data. On the abscissa is the percentile used to derive a threshold for voxel inclusion. The threshold was applied to the standard deviation map across the 12 videos for the perception contrasts (green line) or for the 12 memory contrasts (red line). Thus, when only voxels with a high standard deviation across the memory conditions (but not perception) are used, classification accuracy substantially increases. This shows that there is an asymmetric relation between memory and perception whereby regions highly active during memory are also likely to be active during perception, but not vice versa.

Figure 4. 

Classification of perceptual brain patterns using memory patterns. The plot shows the ability, in terms of classification accuracy, for discriminant solutions trained on memory to classify activation maps derived from the perception data. On the abscissa is the percentile used to derive a threshold for voxel inclusion. The threshold was applied to the standard deviation map across the 12 videos for the perception contrasts (green line) or for the 12 memory contrasts (red line). Thus, when only voxels with a high standard deviation across the memory conditions (but not perception) are used, classification accuracy substantially increases. This shows that there is an asymmetric relation between memory and perception whereby regions highly active during memory are also likely to be active during perception, but not vice versa.

To examine the similarity between memory and perception across the group, we computed a BADA solution in normalized MNI space that included perception, memory, and posttraining perception data sets for every participant. We also computed a “compromise” BADA solution for the perception, memory, and posttraining perception. In this compromise analysis, each data set was first subjected to singular value decomposition and normalized by dividing each datum by the first singular value of its data set. This compromise solution represents the center of gravity of the category factor scores when all three data sets (perception, memory, and postmemory perception) are analyzed together. Each of the three constituent data sets was then projected onto this compromise factor space (see Abdi et al., 2012, for details about this procedure), and their coordinates within this multidimensional space were plotted for the first two components (Figure 5). This plot shows a high degree of representational similarity between all data sets (all pairwise RV coefficients > .84) and further confirms that the structure of memory is very similar to the structure of perception. The brain regions driving this similarity are evident by examining the RV coefficients computed between perception and memory for each of 48 ROIs in the Harvard–Oxford probabilistic atlas of cortical areas (Abdi, 2010b; Kriegeskorte & Bandettini, 2007). As can be seen in Table 5 (ROIs ordered by magnitude of RV coefficient), the brain areas with the highest congruence between perception and memory are in sensory association cortices in the temporal, parietal, and occipital lobes and in inferior and premotor regions of the frontal cortex. The RV coefficients derived from this regional BADA analysis are largely consistent with the voxelwise perception/memory correlation analysis (shown in Figures 2B and 3). Indeed, the correlation between the RV coefficient and the absolute value of the mean correlation in each of the Harvard–Oxford ROIs was .66 (p < .0001).

Figure 5. 

A common discriminant factor space for the perception, memory and posttraining perception data sets. Each video is represented as a Cartesian coordinate in the discriminant factor space. The “+” symbol represents the center of gravity of all the data sets. The arrows point to the location of the coordinate for the perception (green), memory (red), and posttraining perception (blue) data sets, respectively.

Figure 5. 

A common discriminant factor space for the perception, memory and posttraining perception data sets. Each video is represented as a Cartesian coordinate in the discriminant factor space. The “+” symbol represents the center of gravity of all the data sets. The arrows point to the location of the coordinate for the perception (green), memory (red), and posttraining perception (blue) data sets, respectively.

Table 5. 

RV Coefficient between Perception and Memory in Regional BADA Analysis

Harvard–Oxford ROI
RV Coefficient
p
Left
Right
Left
Right
Post. superior temporal gyrus 0.765 0.774 <.0001a <.0001a 
Post. supramarginal gyrus 0.693 0.694 <.0001a <.0001a 
Superior parietal lobule 0.659 0.655 <.0001a <.0001a 
Post. middle temporal gyrus 0.633 0.716 <.0001a <.0001a 
Middle temporal gyrus, temporooccipital 0.658 0.640 <.0001a <.0001a 
Sup. lateral occipital cortex 0.667 0.630 <.0001a <.0001a 
Precentral gyrus 0.638 0.683 <.0001a <.0001a 
Ant. supramarginal gyrus 0.655 0.636 <.0001a <.0001a 
Planum temporale 0.688 0.610 <.0001a <.0001a 
Parietal operculum cortex 0.656 0.591 <.0001a <.0001a 
Postcentral gyrus 0.629 0.614 <.0001a <.0001a 
Inferior frontal gyrus, triangularis 0.607 0.620 <.0001a <.0001a 
Inferior frontal gyrus, opercularis 0.613 0.594 <.0001a <.0001a 
Precuneous cortex 0.588 0.618 <.0001a <.0001a 
Temporal pole 0.602 0.585 <.0001a <.0001a 
Inf. lateral occipital cortex 0.604 0.564 <.0001a <.0001a 
Temporal occipital fusiform cortex 0.600 0.578 <.0001a <.0001a 
Ant. superior temporal gyrus 0.558 0.555 <.0001a <.0001a 
Angular gyrus 0.551 0.579 <.0001a <.0001a 
Frontal orbital cortex 0.601 0.538 <.0001a <.01 
Post. temporal fusiform cortex 0.611 0.532 <.0001a <.01 
Middle frontal gyrus 0.567 0.545 <.0001a <.0001a 
Post. parahippocampal gyrus 0.550 0.548 <.01 <.0001a 
Inferior temporal gyrus, temporooccipital 0.531 0.550 <.01 <.0001a 
Planum polare 0.559 0.530 <.0001a <.0001a 
Lingual gyrus 0.521 0.541 <.01 <.0001a 
Post. cingulate gyrus 0.536 0.533 <.01 <.01 
Insular cortex 0.554 0.517 <.0001a >.05 
Superior frontal gyrus 0.506 0.538 >.05 >.05 
Central opercular cortex 0.552 0.511 <.0001a <.0001a 
Ant. parahippocampal gyrus 0.506 0.521 >.05 >.05 
Supracalcarine cortex 0.510 0.521 <.0001a <.0001a 
 Post. inferior temporal gyrus 0.519 0.509 >.05 >.05 
Ant. middle temporal gyrus 0.472 0.534 <.01 <.0001a 
Frontal pole 0.505 0.500 >.05 >.05 
Hippocampus 0.473 0.517 >.05 <.01 
Occipital fusiform gyrus 0.477 0.489 >.05 <.01 
Ant. cingulate gyrus 0.499 0.473 >.05 >.05 
Cuneal cortex 0.482 0.465 <.01 <.01 
Ant. temporal fusiform cortex 0.457 0.484 <.01 <.01 
Frontal operculum cortex 0.505 0.437 <.0001a <.01 
Ant. inferior temporal gyrus 0.441 0.480 >.05 >.05 
Supplementary motor cortex 0.443 0.456 >.05 >.05 
Occipital pole 0.452 0.442 >.05 >.05 
Heschl's gyrus 0.433 0.457 <.01 <.0001 
Amygdala 0.422 0.467 >.05 >.05 
Intracalcarine cortex 0.435 0.400 >.05 >.05 
Thalamus 0.425 0.401 >.05 >.05 
Globus pallidus 0.420 0.426 >.05 >.05 
Paracingulate gyrus II 0.408 0.403 >.05 >.05 
Paracingulate gyrus I 0.393 0.370 >.05 >.05 
Caudate 0.364 0.361 >.05 >.05 
Frontal medial cortex 0.357 0.346 >.05 >.05 
Putamen 0.354 0.354 >.05 >.05 
Harvard–Oxford ROI
RV Coefficient
p
Left
Right
Left
Right
Post. superior temporal gyrus 0.765 0.774 <.0001a <.0001a 
Post. supramarginal gyrus 0.693 0.694 <.0001a <.0001a 
Superior parietal lobule 0.659 0.655 <.0001a <.0001a 
Post. middle temporal gyrus 0.633 0.716 <.0001a <.0001a 
Middle temporal gyrus, temporooccipital 0.658 0.640 <.0001a <.0001a 
Sup. lateral occipital cortex 0.667 0.630 <.0001a <.0001a 
Precentral gyrus 0.638 0.683 <.0001a <.0001a 
Ant. supramarginal gyrus 0.655 0.636 <.0001a <.0001a 
Planum temporale 0.688 0.610 <.0001a <.0001a 
Parietal operculum cortex 0.656 0.591 <.0001a <.0001a 
Postcentral gyrus 0.629 0.614 <.0001a <.0001a 
Inferior frontal gyrus, triangularis 0.607 0.620 <.0001a <.0001a 
Inferior frontal gyrus, opercularis 0.613 0.594 <.0001a <.0001a 
Precuneous cortex 0.588 0.618 <.0001a <.0001a 
Temporal pole 0.602 0.585 <.0001a <.0001a 
Inf. lateral occipital cortex 0.604 0.564 <.0001a <.0001a 
Temporal occipital fusiform cortex 0.600 0.578 <.0001a <.0001a 
Ant. superior temporal gyrus 0.558 0.555 <.0001a <.0001a 
Angular gyrus 0.551 0.579 <.0001a <.0001a 
Frontal orbital cortex 0.601 0.538 <.0001a <.01 
Post. temporal fusiform cortex 0.611 0.532 <.0001a <.01 
Middle frontal gyrus 0.567 0.545 <.0001a <.0001a 
Post. parahippocampal gyrus 0.550 0.548 <.01 <.0001a 
Inferior temporal gyrus, temporooccipital 0.531 0.550 <.01 <.0001a 
Planum polare 0.559 0.530 <.0001a <.0001a 
Lingual gyrus 0.521 0.541 <.01 <.0001a 
Post. cingulate gyrus 0.536 0.533 <.01 <.01 
Insular cortex 0.554 0.517 <.0001a >.05 
Superior frontal gyrus 0.506 0.538 >.05 >.05 
Central opercular cortex 0.552 0.511 <.0001a <.0001a 
Ant. parahippocampal gyrus 0.506 0.521 >.05 >.05 
Supracalcarine cortex 0.510 0.521 <.0001a <.0001a 
 Post. inferior temporal gyrus 0.519 0.509 >.05 >.05 
Ant. middle temporal gyrus 0.472 0.534 <.01 <.0001a 
Frontal pole 0.505 0.500 >.05 >.05 
Hippocampus 0.473 0.517 >.05 <.01 
Occipital fusiform gyrus 0.477 0.489 >.05 <.01 
Ant. cingulate gyrus 0.499 0.473 >.05 >.05 
Cuneal cortex 0.482 0.465 <.01 <.01 
Ant. temporal fusiform cortex 0.457 0.484 <.01 <.01 
Frontal operculum cortex 0.505 0.437 <.0001a <.01 
Ant. inferior temporal gyrus 0.441 0.480 >.05 >.05 
Supplementary motor cortex 0.443 0.456 >.05 >.05 
Occipital pole 0.452 0.442 >.05 >.05 
Heschl's gyrus 0.433 0.457 <.01 <.0001 
Amygdala 0.422 0.467 >.05 >.05 
Intracalcarine cortex 0.435 0.400 >.05 >.05 
Thalamus 0.425 0.401 >.05 >.05 
Globus pallidus 0.420 0.426 >.05 >.05 
Paracingulate gyrus II 0.408 0.403 >.05 >.05 
Paracingulate gyrus I 0.393 0.370 >.05 >.05 
Caudate 0.364 0.361 >.05 >.05 
Frontal medial cortex 0.357 0.346 >.05 >.05 
Putamen 0.354 0.354 >.05 >.05 

aStatistically significant after Bonferroni correction for multiple comparisons.

Ant. = anterior; Inf. = inferior; Post. = posterior; Sup. = superior.

Stability of Global Perceptual Activation Patterns from before and after Memory Training

For the nine participants that completed the memory training regimen, we were able to compare whole-brain activation patterns for the each video both before and after memory training. Using the contrast map associated with the perception of each video, we computed for each participant the correlation between the pretraining and posttraining perceptual activation maps. The average correlation for all videos and participants was .597. When the average was computed separately for each video, it ranged from a minimum of .420 (“Baboons”) to a maximum of .700 (“Brando”) and had a standard deviation of .097. Correlations between the two perception scans were statistically significant for all videos (all ps < .0001).

DISCUSSION

Our results offer clear evidence for the phenomenological intuition that rich, vivid, and cinematic memories constitute a kind of neural revisitation of the perceptual events that they denote. By using a diverse set of dynamic multimodal stimuli, we were able to mimic the richness and complexity of human perception and memory for episodic events in a laboratory setting. By training participants for many hours to mentally replay these complex and dynamic perceptual events, we were able to maximize the fidelity of the mnemonic representations. We then showed that these vivid memories give rise to patterns of brain activation that are unique and idiosyncratic in a similar fashion to brain patterns evoked by their perceptual precursors.

Previous studies have shown that brain regions active during perception of some class of stimuli are often reactivated during the retrieval of that same class of stimuli (Danker & Anderson, 2010; Stokes, Thompson, Nobre, & Duncan, 2009; Slotnick, 2004; Ruchkin et al., 2003). Demonstrations of reactivation in cued or long-term memory have used only a few classes of stimuli and generally compared only two or three categories at a time (e.g., faces or scenes, words or pictures). This study extends this work by using stimuli that are sufficiently complex that they cannot be classified into simple opposing categories and thus represent unique and temporally extended episodes. Indeed, such stimuli offer the most general and complete test to date for the principle of cortical reactivation during conscious remembering. Instead of focusing in on one or two regions of the brain to test for reactivation of this or that perceptual feature, we formulated our hypothesis in terms of memory reactivation in its totality as a distributed pattern of brain activity. We reasoned that a sufficiently complex perceptual episode would produce a kind of fingerprint in the brain that reliably identifies it, and this was confirmed. We further reasoned that if the pattern of activity that is formed during memory substantially resembles the perceptual fingerprint and can be discriminated from memories for other events, then this would offer the strongest evidence in support of cortical reinstatement. With the caveat that early sensory areas sometimes showed an inverse pattern of reactivation, the hypothesis of global similarity between perception and memory was also unambiguously confirmed.

We also showed that patterns of brain activations elicited by memories are not pixel-perfect replications of perception. Nevertheless, when such images are subjected to a multivariate analysis of their latent structure, the similarities that emerge are striking, with an average shared variance of 91% in the single-subject solutions. Thus, although the pattern of reactivation during vivid memory is partial and involves mostly sensory association and multimodal cortices, there exists a deeper representational structure that links perception and memory. A possible mechanism for the partial reactivation of perceptual brain patterns during mental replay was first suggested by Hebb (1968), who theorized that perceptual memories and mental imagery more generally arise from the top–down activation of higher-order cell assemblies that encode the content of perception at a more abstract or general level. A memory image, in effect, accesses the representation of a perceptual episode at the top of a hierarchical structure, and the extent to which memory resembles perception is related to the depth in which that hierarchy is revisited during the process of reactivation (Meyer & Damasio, 2009; Fuster, 2005). Direct stimulus perception, on the other hand, operates from the bottom–up, starting with activation of the primary cell assemblies and propagating up the hierarchy to second-order and higher-order neuronal assemblies. Thus, the shared abstract structure that emerged from our multivariate analyses likely reflects the pattern reinstatement that occurs at the higher levels of the collection of cell assemblies whose joint activation completely encodes an object of perception.

Some regions of the cerebral cortex show little association or even, in the case of early auditory and visual cortices, an inverse correlation between the activity during perception and memory. For example, the primary visual and auditory cortices showed greater activity (relative to the other videos) during perception but less activity during memory. This intriguing pattern is consistent with a recent finding by Linke, Vicente-Grabovetsky, and Cusack (2011), who showed that the pattern of activity in auditory cortex during the perception of tones was inversely correlated with the pattern that was activated during the STM of those same tones. Thus, the pattern of suppression in auditory cortex was specific to the frequency of the tone stored in STM. Our findings also show that—in contrast to higher-order sensory association and supramodal cortical areas, where memory activation patterns correlate positively with perceptual patterns—a “reversal” occurs from perception to memory whereby stimulus-specific activation during perception becomes stimulus-specific suppression during memory. This phenomenon suggests a potential neural mechanism that, in some circumstances, underlies the ability to distinguish an internal memory representation from the full blown sensory reality of the “here and now.” Indeed, it may be a failure of this mechanism that accounts for pathological reality distortions such as auditory hallucinations, where internally generated signals are confused with genuine sensory stimuli (Dierks et al., 1999). It should be noted, however, that activation in primary visual cortices during mental imagery has been shown in several studies, although these studies have used tasks involving mental visualization of fine-grained details that do not involve spatial relations (Slotnick, Thompson, & Kosslyn, 2005; Kosslyn & Thompson, 2003) or moving images, two prevalent aspects movie stimuli used in the current study.

Our results show that the largest similarity between perception and memory occurs in sensory and motor association areas of the cerebral cortex. Some brain regions that are known to be important for various aspects of memory did not show robust similarity effects in our analyses. For instance, the hippocampus, dorsolateral and anterior regions of the pFC, and the angular gyrus (three regions important for memory retrieval) were not associated with high voxelwise correlations between perception and memory or with high RV coefficients in the regional BADA analysis. Chadwick et al. (2010), however, have shown—using high-resolution fMRI of the medial-temporal lobe—that the patterns of activity in the hippocampus could be used to classify memories for four similar videos, each of which involving a person performing an action in an outdoor city scene. Our analyses, however, did not ask whether there exists information in the memory activation patterns sufficient to discriminate memories for a set of videos, rather we asked whether the activation patterns elicited during the observation of videos were similar to those evoked during memory for these same videos. Thus, our analyses were explicitly aimed at the phenomenon of topographic memory “reactivation,” which is most firmly established for posterior regions of the cerebral cortex in the functional neuroimaging literature. Also, because our analyses were focused on video-specific activation, nonspecific effects of task (i.e., the main effect of mental replay) were subtracted out from each of the video contrasts before any multivariate classification analysis was carried out. An examination of the average activity across all videos during mental replay does indeed show significant activity in the hippocampus and portions of the dorsolateral and anterior pFC (not reported), which is consistent with neuroimaging studies of episodic and autobiographical memory (Spaniol et al., 2009; Svoboda, McKinnon, & Levine, 2006). Thus, in our study, some areas, such as the hippocampus, were generally activated during mental replay but did not show consistently strong patterns of topographic similarity between perception and memory across the selected set of video stimuli. Lastly, it is possible that the extended period of training in our experiment promoted memory consolidation, which according to the standard view (Squire, Stark, & Clark, 2004) may have caused the mnemonic representation to become independent of the hippocampus. This possibility is somewhat mitigated by the above-mentioned finding of a main effect of the mental replay task in the hippocampus.

One might ask whether the findings of the current study, insofar as they are based on memories formed from repeated viewings of a stimulus set and have been highly practiced, generalize to other memory settings. For instance, “episodic memory” is typically defined as a conscious re-experience of a prior event that is unique in place and time (Tulving, 2002). By this definition, our study was not testing “episodic memory” because the objects of memory, having been repeatedly observed and mentally replayed, did not have a unique place in time. Indeed, the type of memory tested in our study falls in a kind of taxonomic lacuna, as it is neither episodic, nor semantic, nor short-term or working memory, nor autobiographical. One might call the task “mental imagery,” but that seems too general a label, as mental imagery need not be constrained by actual events, and can be applied to imaginings of the future, false memories, transformation of mental images, and the like. Neisser (1981) referred to memories distilled from the observation of repeated events as “repisodic memory,” and this term seems to capture, at a descriptive level at least, the sort of memory tested in the present experiment.

We have also often used the generic descriptive terms “perception” and “memory” to refer to the two main cognitive states that have been the subject of our investigations. We use these terms in broadest sense, that is, to denote on the one hand a condition in which the participant receives and processes complex sensory input and on the other hand a condition in which the participant must rely on memory to a evoke an internal approximation of the representations formed as a consequence of that same complex sensory input. In many studies of memory, the terms “encoding” and “retrieval” are used to refer to these two cognitive states (or orientations, or processes). We have avoided these terms primarily because our task did not conform to the standard “encode and retrieve” memory paradigm; we therefore have used the broadest possible terms (perception and memory) and have meant to apply these terms in a descriptive and in some sense atheoretical way.

Despite the difficulties in the theoretical classification of the type of memory tested in the current study, the neurobiological principle of “reactivation” appears to generalize across a variety of memory tasks, including working memory (e.g., Buchsbaum et al., 2005; Druzgal & D'Esposito, 2003), episodic memory (e.g., Wheeler et al., 2000), mental imagery (e.g., Slotnick et al., 2005; Ishai, Haxby, & Ungerleider, 2002; Kosslyn, Thompson, Kim, & Alpert, 1995), and in the present “repisodic” memory task. There is also at least one neuroimaging study that shows that the neural networks that are recruited after autobiographical memory for events rehearsed once, twice, or eight times were stable across repetitions—despite showing some attenuation (Svoboda & Levine, 2009)—and could by reliably distinguished from semantic memories (see also St-Laurent, Abdi, Burianova, & Grady, 2011). Moreover, Svoboda and Levine (2009) showed that ratings of vividness increased with rehearsal number, indicating that retrieval practice can have the effect of enhancing one of the core properties of episodic memory. In the present study, we also showed that accuracy for the chronological component of video memory increased with practice, arguing against the possibility that training serves to reduce—by way of abstraction or semanticization—memory accuracy. Indeed, the demonstrated similarity in brain activation between initial stimulus perception and mental replay after extensive training is prima facie evidence that mnemonic abstraction, schematization, or other transformative processes did not sufficiently distort the memory representation so as to preclude our ability to show a high degree of shared similarity structure between brain activation patterns at perception and memory.

Acknowledgments

This research was funded by grants from the Canadian Institutes of Health Research and the Natural Sciences and Research Council of Canada. We thank Jeremy Young and Ashley Bondad for help with task programming and subject recruitment.

Reprint requests should be sent to Bradley R. Buchsbaum, Rotman Research Institute, Baycrest Hospital, 3560 Bathurst St., Toronto, ON M6A 2E1, or via e-mail: bbuchsbaum@rotman-baycrest.on.ca.

REFERENCES

REFERENCES
Abdi
,
H.
(
2010a
).
Barycentric discriminant analysis (BADIA).
In N. J. Salkind, D. M. Dougherty, & B. Frey (Eds.)
,
Encyclopedia of research design
(pp.
64
75
). Thousand Oaks, CA: Sage.
Abdi
,
H.
(
2010b
).
Congruence: Congruence coefficient, RV coefficient, and Mantel coefficient.
In N. J. Salkind, D. M. Dougherty, & B. Frey (Eds.)
,
Encyclopedia of research design
(pp.
222
229
). Thousand Oaks, CA: Sage.
Abdi
,
H.
,
Williams
,
L. J.
,
Gobbini
,
M. I.
,
Dunlop
,
J. P.
, &
Haxby
,
J. V.
(
2012
).
Multiple subject barycentric discriminant analysis (MUSUBADA): How to assign scans to categories without using spatial normalization.
Computational and Mathematical Methods in Medicine
,
doi: 10.1155/2012/634165
.
Abdi
,
H.
,
Williams
,
L. J.
,
Valentin
,
D.
, &
Bennani-Dosse
,
M.
(
2012
).
STATIS and DISTATIS: Optimum multi-table principal component analysis and three-way metric multidimensional scaling.
Wiley Interdisciplinary Reviews: Computational Statistics
,
5
,
124
167
.
Avants
,
B. B.
,
Epstein
,
C. L.
,
Grossman
,
M.
, &
Gee
,
J. C.
(
2008
).
Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain.
Medical Image Analysis
,
12
,
26
41
.
Baddeley
,
A.
, &
Wilson
,
B. A.
(
1993
).
A case of word deafness with preserved span: Implications for the structure and function of short-term memory.
Cortex
,
29
,
741
748
.
Buchsbaum
,
B. R.
,
Olsen
,
R. K.
,
Koch
,
P.
, &
Berman
,
K. F.
(
2005
).
Human dorsal and ventral auditory streams subserve rehearsal-based and echoic processes during verbal working memory.
Neuron
,
48
,
687
697
.
Chadwick
,
M. J.
,
Hassabis
,
D.
, &
Maguire
,
E. A.
(
2011
).
Decoding overlapping memories in the medial temporal lobes using high-resolution fMRI.
Learning and Memory
,
18
,
742
746
.
Chadwick
,
M. J.
,
Hassabis
,
D.
,
Weiskopf
,
N.
, &
Maguire
,
E. A.
(
2010
).
Decoding individual episodic memory traces in the human hippocampus.
Current Biology
,
20
,
544
547
.
Cox
,
R. W.
(
1996
).
AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages.
Computers in Biomedical Research
,
29
,
162
173
.
Craik
,
F. I.
(
2002
).
Levels of processing: Past, present, and future?
Memory
,
10
,
305
318
.
Damasio
,
A. R.
(
1989
).
Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition.
Cognition
,
33
,
25
62
.
Danker
,
J. F.
, &
Anderson
,
J. R.
(
2010
).
The ghosts of brain states past: Remembering reactivates the brain regions engaged during encoding.
Psychological Bulletin
,
136
,
87
102
.
Dierks
,
T.
,
Linden
,
D. E.
,
Jandl
,
M.
,
Formisano
,
E.
,
Goebel
,
R.
,
Lanfermann
,
H.
,
et al
(
1999
).
Activation of Heschl's gyrus during auditory hallucinations.
Neuron
,
22
,
615
621
.
Druzgal
,
T. J.
, &
D'Esposito
,
M.
(
2003
).
Dissecting contributions of prefrontal cortex and fusiform face area to face working memory.
Journal of Cognitive Neuroscience
,
15
,
771
784
.
Fuster
,
J.
(
2005
).
Cortex and mind: Unifying cognition.
New York
:
Oxford University Press
.
Genovese
,
C. R.
,
Lazar
,
N. A.
, &
Nichols
,
T.
(
2002
).
Thresholding of statistical maps in functional neuroimaging using the false discovery rate.
Neuroimage
,
15
,
870
878
.
Hassabis
,
D.
,
Kumaran
,
D.
, &
Maguire
,
E. A.
(
2007
).
Using imagination to understand the neural basis of episodic memory.
Journal of Neuroscience
,
27
,
14365
14374
.
Hasson
,
U.
,
Nir
,
Y.
,
Levy
,
I.
,
Fuhrmann
,
G.
, &
Malach
,
R.
(
2004
).
Intersubject synchronization of cortical activity during natural vision.
Science
,
303
,
1634
1640
.
Hastie
,
T.
,
Tibshirani
,
R.
, &
Friedman
,
J.
(
2001
).
The elements of statistical learning.
New York
:
Springer
-Verlag.
Haxby
,
J. V.
,
Gobbini
,
M. I.
,
Furey
,
M. L.
,
Ishai
,
A.
,
Schouten
,
J. L.
, &
Pietrini
,
P.
(
2001
).
Distributed and overlapping representations of faces and objects in ventral temporal cortex.
Science
,
293
,
2425
2430
.
Hebb
,
D. O.
(
1968
).
Concerning imagery.
Psychological Review
,
75
,
466
477
.
Ishai
,
A.
,
Haxby
,
J. V.
, &
Ungerleider
,
L. G.
(
2002
).
Visual imagery of famous faces: Effects of memory and attention revealed by fMRI.
Neuroimage
,
17
,
1729
1741
.
Ishai
,
A.
,
Ungerleider
,
L. G.
, &
Haxby
,
J. V.
(
2000
).
Distributed neural systems for the generation of visual images.
Neuron
,
28
,
979
990
.
Kosslyn
,
S. M.
,
Ganis
,
G.
, &
Thompson
,
W. L.
(
2001
).
Neural foundations of imagery.
Nature Reviews Neuroscience
,
2
,
635
642
.
Kosslyn
,
S. M.
, &
Thompson
,
W. L.
(
2003
).
When is early visual cortex activated during visual mental imagery?
Psychological Bulletin
,
129
,
723
746
.
Kosslyn
,
S. M.
,
Thompson
,
W. L.
,
Kim
,
I. J.
, &
Alpert
,
N. M.
(
1995
).
Topographical representations of mental images in primary visual cortex.
Nature
,
378
,
496
498
.
Kriegeskorte
,
N.
, &
Bandettini
,
P.
(
2007
).
Combining the tools: Activation- and information-based fMRI analysis.
Neuroimage
,
38
,
666
668
.
Kriegeskorte
,
N.
,
Mur
,
M.
, &
Bandettini
,
P.
(
2008
).
Representational similarity analysis − Connecting the branches of systems neuroscience.
Frontiers in Systems Neuroscience
,
2
,
4
.
Linke
,
A. C.
,
Vicente-Grabovetsky
,
A.
, &
Cusack
,
R.
(
2011
).
Stimulus-specific suppression preserves information in auditory short-term memory.
Proceedings of the National Academy of Sciences, U.S.A.
,
108
,
12961
12966
.
Meyer
,
K.
, &
Damasio
,
A.
(
2009
).
Convergence and divergence in a neural architecture for recognition and memory.
Trends in Neurosciences
,
32
,
376
382
.
Neisser
,
U.
(
1981
).
John Dean's memory: A case study.
Cognition
,
9
,
1
22
.
Nichols
,
T.
,
Brett
,
M.
,
Andersson
,
J.
,
Wager
,
T.
, &
Poline
,
J. B.
(
2005
).
Valid conjunction inference with the minimum statistic.
Neuroimage
,
25
,
653
660
.
Nyberg
,
L.
,
Habib
,
R.
,
McIntosh
,
A. R.
, &
Tulving
,
E.
(
2000
).
Reactivation of encoding-related brain activity during memory retrieval.
Proceedings of the National Academy of Sciences, U.S.A.
,
97
,
11120
11124
.
O'Toole
,
A. J.
,
Jiang
,
F.
,
Abdi
,
H.
,
Penard
,
N.
,
Dunlop
,
J. P.
, &
Parent
,
M. A.
(
2007
).
Theoretical, statistical, and practical perspectives on pattern-based classification approaches to the analysis of functional neuroimaging data.
Journal of Cognitive Neuroscience
,
19
,
1735
1752
.
Place
,
U. T.
(
1956
).
Is consciousness a brain process?
British Journal of Clinical Psychology
,
47
,
44
50
.
Polyn
,
S. M.
,
Natu
,
V. S.
,
Cohen
,
J. D.
, &
Norman
,
K. A.
(
2005
).
Category-specific cortical activity precedes retrieval during memory search.
Science
,
310
,
1963
1966
.
Postle
,
B. R.
(
2006
).
Working memory as an emergent property of the mind and brain.
Neuroscience
,
139
,
23
38
.
Rama
,
P.
, &
Courtney
,
S. M.
(
2005
).
Functional topography of working memory for face or voice identity.
Neuroimage
,
24
,
224
234
.
Ranganath
,
C.
,
Cohen
,
M. X.
,
Dam
,
C.
, &
D'Esposito
,
M.
(
2004
).
Inferior temporal, prefrontal, and hippocampal contributions to visual working memory maintenance and associative memory retrieval.
Journal of Neuroscience
,
24
,
3917
3925
.
Rissman
,
J.
, &
Wagner
,
A. D.
(
2012
).
Distributed representations in memory: Insights from functional brain imaging.
Annual Review of Psychology
,
63
,
101
128
.
Ruchkin
,
D. S.
,
Grafman
,
J.
,
Cameron
,
K.
, &
Berndt
,
R. S.
(
2003
).
Working memory retention systems: A state of activated long-term memory.
Behavioral and Brain Sciences
,
26
,
709–728; discussion 728-777
.
Schumacher
,
E. H.
,
Lauber
,
E.
,
Awh
,
E.
,
Jonides
,
J.
,
Smith
,
E. E.
, &
Koeppe
,
R. A.
(
1996
).
PET evidence for an amodal verbal working memory system.
Neuroimage
,
3
,
79
88
.
Shallice
,
T.
(
990
).
The impairment of auditory-verbal short-term storage.
In G. Vallar & T. Shallice (Eds.)
,
Neuropsychological impairments of short-term memory
(pp.
11
53
).
Cambridge
:
Cambridge University Press
.
Slotnick
,
S. D.
(
2004
).
Visual memory and visual perception recruit common neural substrates.
Behavioral and Cognitive Neuroscience
,
3
,
207
221
.
Slotnick
,
S. D.
,
Thompson
,
W. L.
, &
Kosslyn
,
S. M.
(
2005
).
Visual mental imagery induces retinotopically organized activation of early visual areas.
Cerebral Cortex
,
15
,
1570
1583
.
Spaniol
,
J.
,
Davidson
,
P. S.
,
Kim
,
A. S.
,
Han
,
H.
,
Moscovitch
,
M.
, &
Grady
,
C. L.
(
2009
).
Event-related fMRI studies of episodic encoding and retrieval: Meta-analyses using activation likelihood estimation.
Neuropsychologia
,
47
,
1765
1779
.
Squire
,
L. R.
,
Stark
,
C. E.
, &
Clark
,
R. E.
(
2004
).
The medial temporal lobe.
Annual Review of Neuroscience
,
27
,
279
306
.
St. Laurent
,
M.
,
Abdi
,
H.
,
Burianova
,
H.
, &
Grady
,
C. L.
(
2011
).
Influence of aging on the neural correlates of autobiographical, episodic, and semantic memory retrieval.
Journal of Cognitive Neuroscience
,
23
,
4150
4163
.
Stokes
,
M.
,
Thompson
,
R.
,
Nobre
,
A. C.
, &
Duncan
,
J.
(
2009
).
Shape-specific preparatory activity mediates attention to targets in human visual cortex.
Proceedings of the National Academy of Sciences, U.S.A.
,
106
,
19569
19574
.
Strother
,
S. C.
,
Anderson
,
J.
,
Hansen
,
L. K.
,
Kjems
,
U.
,
Kustra
,
R.
,
Sidtis
,
J.
,
et al
(
2002
).
The quantitative evaluation of functional neuroimaging experiments: The NPAIRS data analysis framework.
Neuroimage
,
15
,
747
771
.
Svoboda
,
E.
, &
Levine
,
B.
(
2009
).
The effects of rehearsal on the functional neuroanatomy of episodic autobiographical and semantic remembering: A functional magnetic resonance imaging study.
Journal of Neuroscience
,
29
,
3073
3082
.
Svoboda
,
E.
,
McKinnon
,
M. C.
, &
Levine
,
B.
(
2006
).
The functional neuroanatomy of autobiographical memory: A meta-analysis.
Neuropsychologia
,
44
,
2189
2208
.
Tulving
,
E.
(
2002
).
Episodic memory: From mind to brain.
Annual Review of Psychology
,
53
,
1
25
.
Warrington
,
E.
, &
Shallice
,
T.
(
1969
).
Selective impairment of auditory verbal short-term memory.
Brain
,
92
,
885
896
.
Wheeler
,
M. E.
,
Petersen
,
S. E.
, &
Buckner
,
R. L.
(
2000
).
Memory's echo: Vivid remembering reactivates sensory-specific cortex.
Proceedings of the National Academy of Sciences, U.S.A.
,
97
,
11125
11129
.