Abstract
Episodic memories are not static but can change on the basis of new experiences, potentially allowing us to make valid predictions in the face of an ever-changing environment. Recent research has identified prediction errors during memory retrieval as a possible trigger for such changes. In this study, we used modified episodic cues to investigate whether different types of mnemonic prediction errors modulate brain activity and subsequent memory performance. Participants encoded episodes that consisted of short toy stories. During a subsequent fMRI session, participants were presented videos showing the original episodes, or slightly modified versions thereof. In modified videos, either the order of two subsequent action steps was changed or an object was exchanged for another. Content modifications recruited parietal, temporo-occipital, and parahippocampal areas reflecting the processing of the new object information. In contrast, structure modifications elicited activation in right dorsal premotor, posterior temporal, and parietal areas, reflecting the processing of new sequence information. In a post-fMRI memory test, the participants' tendency to accept modified episodes as originally encoded increased significantly when they had been presented modified versions already during the fMRI session. After experiencing modifications, especially those of the episodes' structure, the recognition of originally encoded episodes was impaired as well. Our study sheds light onto the neural processing of different types of episodic prediction errors and their influence on subsequent memory recall.
INTRODUCTION
Episodic memories enable us to vividly relive events that we experienced at some point in our personal life (Tulving, 2002). However, there is evidence that they are not always veridical reconstructions of our past (Lee, Nader, & Schiller, 2017; Scully, Napper, & Hupbach, 2017; Nader, 2015; Nader & Einarsson, 2010). Situations we encounter in everyday life are usually not exactly the same as those we experienced before. So, there is always a certain discrepancy between our expectations, which we derive from our memories, and the new events we experience. According to the predictive coding framework, this discrepancy leads to a prediction error (Reichardt, Polner, & Simor, 2020; Barto, Mirolli, & Baldassarre, 2013). Prediction errors serve as bottom–up learning signals that allow us to adapt our internal predictive models to an ever-changing environment to maintain valid predictions in the long run (Schubotz, 2015; Friston & Kiebel, 2009; Friston, 2005). According to this view, it is adaptive that memories are modified in favor of valid internal models informed and updated by later experiences (Fernández, Boccia, & Pedreira, 2016; Exton-McGuinness, Lee, & Reichelt, 2015). Evidence accumulates that mnemonic prediction errors are important drivers of memory change (Sinclair & Barense, 2019), and researchers recently begun to address the question how mnemonic prediction errors are processed by the brain (e.g., Bein, Duncan, & Davachi, 2020; Kim, Lewis-Peacock, Norman, & Turk-Browne, 2014).
The aim of this study was to characterize neural responses to different types of mnemonic prediction errors during episodic retrieval by targeting two basic types of episodic memory information: either their content (“what”) or their structure (“when”; cf. Griffiths, Dickinson, & Clayton, 1999). To do so, we adapted a previously developed episodic cueing paradigm (Jainta et al., 2022; Schiffer, Ahlheim, Ulrichs, & Schubotz, 2013; Schiffer, Ahlheim, Wurm, & Schubotz, 2012). After encoding short episodes from videos and consolidating memories in two further retrieval sessions, participants went through a fMRI session and were either presented original episode videos or slightly modified versions thereof. To create the latter, a subset of videos was manipulated with regard to the occurrence of an object (content modification) or the order of two consecutive action steps (structure modification) to elicit different types of mnemonic prediction errors (see Figure 1 for an example). In a post-fMRI memory test, participants' memory for original and modified episodes was probed.
Example of an original episode and its modified versions, shown by the sequence of the main event steps. Twenty-four stories existed in three different versions each: an original, a structure modification, and a content modification. For the structure modifications, two adjacent action steps were switched compared with the original. In this example, the original shows the blonde woman join the scene before the guinea pig is positioned on the sales counter; in the structure modification, the guinea pig appears before the blonde woman (red circles). For the content modifications, an object was exchanged compared with the original (here: tortoise instead of guinea pig on the sales counter in Step 5). Note that in the fMRI experiment, each participant was only presented with one of the three versions of a story. We do not reproduce photos of our stimulus material because it is copyrighted material (PLAYMOBIL figures); instead, we provide schematic images.
Example of an original episode and its modified versions, shown by the sequence of the main event steps. Twenty-four stories existed in three different versions each: an original, a structure modification, and a content modification. For the structure modifications, two adjacent action steps were switched compared with the original. In this example, the original shows the blonde woman join the scene before the guinea pig is positioned on the sales counter; in the structure modification, the guinea pig appears before the blonde woman (red circles). For the content modifications, an object was exchanged compared with the original (here: tortoise instead of guinea pig on the sales counter in Step 5). Note that in the fMRI experiment, each participant was only presented with one of the three versions of a story. We do not reproduce photos of our stimulus material because it is copyrighted material (PLAYMOBIL figures); instead, we provide schematic images.
Although today many agree that prediction errors drive memory modification during episodic retrieval (Barron, Auksztulewicz, & Friston, 2020; Fernández et al., 2016; Kim et al., 2014), there is no unifying model from which neuroanatomical hypotheses can be derived. Based on existing studies, we expected that some regions might be engaged in episodic prediction errors in general, whereas others would be engaged only in content or structure prediction errors.
As to the former, the medial frontal cortex may serve more general control over consolidation and retrieval of long-term memories (Peters, David, Marcus, & Smith, 2013; Euston, Gruber, & McNaughton, 2012). Furthermore, the hippocampus is regarded a core structure of episodic memory (Horner & Doeller, 2017; Stachenfeld, Botvinick, & Gershman, 2017; Maguire, Intraub, & Mullally, 2016) and responds to mnemonic prediction errors (Bein et al., 2020; Long, Lee, & Kuhl, 2016). In addition to these common neural responses to episodic surprise, structure and content episodic modifications were expected to engage different brain regions. Structure modifications should elevate activity in premotor areas because of their central role in sequential order processing (Schubotz, 2004). More specifically, dorsal premotor and adjacent prefrontal sites along the superior frontal sulcus (SFS; dorsal premotor cortex [PMd]) were found for stepwise ordinal linking of individual action or event steps, as required in different predictive tasks (Pomp et al., 2021; Hrkać, Wurm, & Schubotz, 2014; Schubotz, Korb, Schiffer, Stadler, & von Cramon, 2012; Stadler et al., 2011; Tamber-Rosenau, Esterman, Chiu, & Yantis, 2011; Kurby & Zacks, 2008). By contrast, content modifications were expected to engage areas related to object processing, including lateral occipitotemporal cortex (OTC; Lingnau & Downing, 2015), anterior intraparietal sulcus (IPS; Schubotz, Wurm, Wittmann, & von Cramon, 2014; Creem-Regehr, 2009), and fusiform gyrus (FG; Reber, Gitelman, Parrish, & Mesulam, 2005).
If new content and/or structure information induced updating of the original predictive model during fMRI, as expected, this should also reduce memory accuracy in a post-fMRI memory test (Jainta et al., 2022; Schiffer et al., 2012, 2013). We thus expected a weakening of the original episodic memory, that is, false rejections of original videos as new, and/or the creation of alternative episode representations, that is, false acceptances of modified videos as originals.
METHODS
Participants
Forty-five women took part in the study. Participants had (corrected-to-) normal vision, were native German speakers, and were right-handed as assessed by the Edinburgh Handedness Inventory (Oldfield, 1971). As in our previous study (Jainta et al., 2022), participants were all female to achieve a good match between the hands in the videos and the hands of the participants. They reported no history of neurological or psychiatric disorders or substance abuse. Four participants started the experiment but did not finish, either because of technical problems during the second retrieval session (three participants) or personal reasons (one participant). Data from five additional participants were excluded from analyses because of the incorrect presentation of video stimuli during the fMRI session (one participant) and increased movement during the fMRI session (four participants, approximately 5-mm movement). Consequently, 36 participants were part of the final sample (M = 22 years, SD = 2.78 years, range = 18–30 years). Similar sample sizes have yielded stable results in our previous work (e.g., Jainta et al., 2022; Pomp et al., 2021; El-Sourani, Trempler, Wurm, Fink, & Schubotz, 2019). Participants received course credits or money for their participation and gave written informed consent to participate in this study. The study was conducted in accordance with the Declaration of Helsinki and approved by the local ethics committee of the University of Münster.
Stimuli
We used the same set of videos as previously reported (Jainta et al., 2022; available upon request at https://www.uni-muenster.de/IVV5PSY/AvicomSrv/). These were 78 short videos (duration = 8.80–17.88 sec, M = 12.71 sec) of stories that were played with PLAYMOBIL toys, showing only the toys and hands and underarms of an actress. Stories comprised six to nine action steps (M = 7.4 steps) and 4–14 separable objects (M = 6.93 objects), such as characters, animals, vehicles, and tools. The same object appeared in only one of the stories.
Stories were filmed from above with a digital single-lens reflex camera (Nikon D5300), which was centrally mounted above the table and faced straight down. Matte white paper served as a base. A frame of 47.5 cm × 28 cm was taped on the paper, congruent with the section captured by the camera (in the following referred to as camera frame). Objects that were needed for a particular story were positioned next to the camera frame and were only moved into view in the moment at which they appeared in the story. During filming, the actress wore a black pullover and black rubber gloves. To facilitate future imitation from demo videos, the back of the right hand was marked with a yellow dot (Franz, Ford, & Werner, 2007). Video material was edited using Adobe Premiere Pro CC (Adobe Systems Software, Version 12.1.2). All videos had a frame of size 1920 × 1080 pixels and a frame rate of 25 frames per second. Videos started with seven frames showing only background and ended after seven frames showing the final toy constellation. Throughout the experiment, videos were presented at a visual angle of approximately 7.3° × 13° using Presentation software (Version 20.3 02.25.19, NeuroBehavioral Systems).
On the basis of two pilot studies, we chose 24 out of originally 30 stories for our stimulus set. Stories were excluded when they were particularly difficult to imitate or describe. One of the 30 stories was excluded because of low memorability as indicated by low performance in a signal detection task.
The 24 final stories existed in three different versions each: (1) an original version as encoded by the participants, (2) a version in which two adjacent action steps were switched (structure modification), and (3) another variation of the original video in which one object was exchanged (content modification). Story scripts were created by five experimenters who all had to agree that the original story and modifications thereof were semantically valid (within a toy world) and that modifications did not change the overall outcome of the story. For creating videos with modifications, the respective stories were played and filmed again exactly the way as for the original video. The only aspect that differed between original and modified versions was a single change of either the order of two action steps (i.e., one transition out of 7.33 transitions, on average, for structure modifications) or one object (i.e., one object out of 6.95 objects, on average, for content modifications).
Modifications were never introduced in the first two action steps so that the beginning of a video served as a cue for prediction. Furthermore, no modifications were introduced in the last two action steps, either. The exact time point of the modification in each video was determined by identifying the video frame that diverged from the original version. For an example of an episode and its modified versions, see Figure 1.
Six other stories were used in one version only. Four of them were presented for the first time in the fMRI session, we refer to them as novel episodes in the following. The two remaining videos were only used for practice and did not appear in the fMRI experiment and memory test.
Procedure
Encoding
Encoding sessions were conducted in a computer laboratory at the Department of Psychology at the University of Münster and followed our previously reported protocol, with some modifications (Jainta et al., 2022). The encoding consisted of two sessions that took place on two consecutive days and lasted about 2 and 1.5 hr, respectively. During each of the two sessions, participants encoded half of the episodes. We chose to split the training over 2 days to avoid fatigue or a decrease in motivation because of the relatively long duration of the task.
The 24 demo videos were organized in four subsets, containing six videos each, balanced for the number of action steps (A1, A2, B1, B2). On each day, participants encoded one A and one B subset. This means that each participant encoded each video either during Session 1 or during Session 2; the same video was not encoded on both days. Which subsets were trained in which session was balanced over participants. The order in which episodes were encoded was randomized for each participant. The first session started with two practice videos to familiarize participants with the task.
During encoding, participants sat at the same setup that had been used for filming the stimulus material and likewise wore a black pullover and gloves with a yellow dot on the right hand, so that they could be filmed while playing the stories themselves. The experimenter sat opposite of the participant, supervising the performance. For each story, the toys were positioned next to the camera frame, following the same arrangement as used while creating the stimulus material (Figure 2). Each episode video was presented 3 times from the first-person perspective. Then, participants had to imitate each story correctly 3 times. After imitation, participants had to deliver a detailed description of the story to ensure that they understood it correctly and had paid attention to all objects involved. If participants made a mistake during an imitation or description trial, they were immediately interrupted by the experimenter to avoid encoding of incorrect scripts. They would then start over with a new imitation/description attempt. On average, participants only performed one incorrect imitation attempt (M = 1.04, SD = 0.47).
Encoding setup. During encoding, participants imitated toy stories from demo videos, while sitting at the filming setup. Their performance was monitored by the experimenter.
Encoding setup. During encoding, participants imitated toy stories from demo videos, while sitting at the filming setup. Their performance was monitored by the experimenter.
Retrieval Sessions
To further consolidate episodic memories, participants went through two additional sessions during which they completed an active retrieval task of before encoded episodes. Active retrieval is known to aid memory consolidation and improve retention (Rowland, 2014). The first consolidation session took place on the day after the second encoding session. The second session was conducted approximately 1 week later (range = 4–8 days; M = 6.36 days, SD = 0.93 days).
Participants always watched the first two steps of a demo video. Then, the video stopped and a question was displayed below the still video frame, which either read “Left?” or “Right?”. The participants' task was to visualize the rest of the story from memory the way they had performed it and then answer how many steps of the entire story had been performed with the left or right hand, respectively. They answered by pressing a number key (0–9) on their keyboard. Upon response delivery, the video played until the end and participants were instructed to carefully watch the video to self-check their response. Afterward, written feedback (“correct,” “incorrect”) was provided for 1.5 sec. When feedback is included in retrieval tasks, consolidation has been shown to occur independent of initial retrieval success (Rowland, 2014; Roediger & Butler, 2011). We chose this task because we wanted to encourage active retrieval of the encoded episodes without laying a special focus on aspects that would be modified during the fMRI session. Importantly, the number of steps that were conducted with the left or right hand did not change in videos containing either type of modification. The task was self-paced and started with two practice videos during which the experimenter carefully checked whether the participant had understood the task correctly.
During the retrieval sessions, we established the two experimental factors consolidation Times and Schedule. To this end, half of all episodes were consolidated 2 times in total, the other half 8 times (factor Times). Furthermore, half of the stories were consolidated during both separate sessions, that is, in a spaced manner, while the other half of the stories were consolidated only in Session 2, that is, in a massed manner (factor Schedule). Participants were explicitly instructed to always visualize the story and not just remember previous responses they gave. Each type of question was presented equally often after each video and per session. At the end of Session 2, participants went through a short practice (four video trials, four question trials, one null event) of the task they would conduct during the fMRI scan.
The two described experimental factors consolidation Times (2, 8) and Schedule (spaced, massed) are not further addressed in this article, as they are central for a companion paper (Siestrup, Jainta, Trempler, Cheng, & Schubotz, in preparation) describing the influence of different consolidation strategies on brain activation during episodic recall. Both factors were balanced with respect to the factors reported here, so we can exclude any confounding effects (fully crossed design). Correct answer rates in the retrieval task did not differ significantly between episodes, which were later presented in the original (ori), structure modified (str), or content-modified (con) version (Mori = .854 ± .013; Mstr = .846 ± .012; Mcon = .852 ± .008; F(2, 70) = .314, p = .732, ηp2 = .009), so that we can rule out that any confound was introduced through the retrieval sessions.
fMRI Session
The fMRI session took place approximately 1 week after the second retrieval session (range = 6–13 days; M = 7.69 days, SD = 1.31 days) and was conducted as previously described (Jainta et al., 2022). Participants were told that videos of themselves playing the stories would be presented in the fMRI session. Although participants had actually been filmed during encoding, these videos were not used during the fMRI experiment. This was only a cover story to elevate personal identification with the videos to benefit episode reactivation. We previously confirmed in a pilot study that this cover story works as intended and already applied it successfully in our previous fMRI study (Jainta et al., 2022). Participants were fully debriefed after completion of the study.
During the fMRI session, participants were presented with original and modified videos reminiscent of the previously encoded episodes. Importantly, each video was only shown in the original or one divergent version. Following a previously used paradigm (Schiffer et al., 2012, 2013), modified and original episodes were presented repeatedly to simulate the natural circumstances that potentially foster memory modification, that is, updating of internal models because of increasing evidence for the validity of an alternative. Thus, eight videos were repeatedly presented in the original version; eight included a structure modification; and eight, a content modification. Which stories belonged to which conditions varied between participants. In addition, four novel stories were included in the fMRI session.
The fMRI experiment consisted of six blocks, each containing the 24 videos reminiscent of the previously encoded episodes. Consequently, each video was presented 6 times over the course of the session. Within blocks, videos were presented in pseudorandom order so that transition probabilities between conditions were balanced. In addition, each block contained three null events during which only a fixation cross was presented (duration: 7–10 sec). Furthermore, each novel video was presented once per block. Therefore, the whole experiment contained 18 null events and 24 novel video trials. Participants were not informed about the block structure of the experiment.
Participants were instructed to attentively watch the presented videos. They were told that after some videos, a short description would be presented (e.g., “Rescuing princess”) that either matched or did not match the story shown in the video (question trials). The task was to either accept or reject the description by pressing one of two buttons on a response box with the right index or middle finger, respectively. This type of task has been used successfully before to focus participants' attention on complex video stimuli (Jainta et al., 2022; El-Sourani et al., 2019). Importantly, neither type of modification influenced the overall outcome of episodes so that all descriptions used as questions were valid for all episode versions. Questions never highlighted any type of modification (Figure 3). Throughout the entire experiment, each story was once followed by a matching description and once by a nonmatching description, resulting in a total number of 56 question trials in the experiment. Each block contained 9–10 question trials and, per block, approximately 50% of descriptions were to be accepted, and 50% were to be rejected. The question was presented for a maximum of 3 sec or until participants responded. Upon response delivery, participants received a 1-sec written feedback whether they answered correctly, incorrectly, or too late, in case no response was given. Participants were naive with regard to this distribution of question trials.
Schematic depiction of task during fMRI session. Video trials consisted of a variable jitter (0, 0.5, 1, or 1.5 sec of fixation), a video showing a toy story (ca. 9–18 sec) and a 2-sec interstimulus interval (fixation). Question trials included a variable jitter, a question regarding the story shown in the preceding video (maximally 3 sec long or terminated by response), and a 2-sec interstimulus interval. The interstimulus interval after question trials was divided into a 1-sec feedback (“correct,” “incorrect,” “too late”) and a 1-sec fixation. Aside from the question, it was depicted which button should be pressed to accept (left, green) or reject (right, red) the description. For each modified video, we determined the exact video frame during which the modification occurred (time of modification), which we used to precisely model modification-related brain activation. For original and novel videos, comparable time points were chosen.
Schematic depiction of task during fMRI session. Video trials consisted of a variable jitter (0, 0.5, 1, or 1.5 sec of fixation), a video showing a toy story (ca. 9–18 sec) and a 2-sec interstimulus interval (fixation). Question trials included a variable jitter, a question regarding the story shown in the preceding video (maximally 3 sec long or terminated by response), and a 2-sec interstimulus interval. The interstimulus interval after question trials was divided into a 1-sec feedback (“correct,” “incorrect,” “too late”) and a 1-sec fixation. Aside from the question, it was depicted which button should be pressed to accept (left, green) or reject (right, red) the description. For each modified video, we determined the exact video frame during which the modification occurred (time of modification), which we used to precisely model modification-related brain activation. For original and novel videos, comparable time points were chosen.
Between trials, a fixation cross was presented for a duration of 2 sec (1 sec after question trials) to serve as an interstimulus interval. Before each trial, a variable jitter of 0, 0.5, 1, or 1.5 sec of fixation was added for enhancement of the temporal resolution of the BOLD response (Figure 3). In total, the fMRI task had a duration of approximately 48 min.
Post-fMRI Memory Test
Immediately after the fMRI session, participants completed an explicit memory test as described previously (Jainta et al., 2022). Importantly, encoding occurred incidentally, as participants were not informed beforehand that their memory for episodes would be tested.
Participants were seated in a separate room in front of a laptop and instructed to remember their encoding sessions 2 weeks prior during which they had played the stories themselves. They were presented all stories that they had seen in the fMRI session in two different versions. More precisely, when modified videos had been presented during the fMRI session, these modified videos were presented again during the memory test and additionally each story was shown in the original version. When original episode videos had been presented during the fMRI scan, these original videos were presented again in the memory test and, additionally, each story was shown in a modified version, either containing a structure modification in half of the cases or a content modification.
The participants' task was to rate after each video whether they knew this exact episode from the encoding sessions, using a Likert scale including 1 (yes), 2 (rather yes), 3 (rather no), and 4 (no), by pressing one out of four marked keys on the laptop's keyboard. Similar rating schemes have previously been used in memory research (Jainta et al., 2022; Kim et al., 2014). Response time was not restricted, but participants were instructed to respond quickly and intuitively. Videos were presented in a pseudorandomized order, so that half of the stories (of each experimental condition) were first presented in their original version followed by a modified version and vice versa. Novel videos were shown twice in the same version, so that, in total, the memory test comprised 56 video trials. The completion of the task took approximately 15 min.
MRI Data Acquisition and Preprocessing
MRI scans were conducted with a 3-Tesla Siemens Magnetom Prisma MR tomograph using a 20-channel head coil. Participants lay supine on the scanner bed with their right index and middle finger positioned on the two appropriate buttons on a response box. Head, arm, and hand movements were minimized by tight fixation with form-fitting cushions. Participants were provided with earplugs and headphones to attenuate scanner noise. Stimuli were projected on a screen that the participants saw via an individually adjusted mirror, which was mounted on the head coil.
High resolution T1-weighted anatomical images were obtained with a 3-D multiplanar rapidly acquired gradient echo sequence before functional imaging. One hundred ninety-two slices with a thickness of 1 mm were acquired, using a repetition time of 2130 msec, an echo time of 2.28 msec, a flip angle of 8°, and a field of view of 256 × 256 mm2. Functional images of the whole brain were acquired in interleaved order along the anterior commissure–posterior commissure plane using a gradient-echo EPI sequence to measure BOLD contrast. Thirty-three axial slices with a thickness of 3 mm (voxel size 3 mm3) were obtained, using a repetition time of 2000 msec, an echo time of 30 msec, a field of view of 192 × 192 mm2, and a flip angle of 90°.
Processing of imaging data was conducted with SPM12 (Wellcome Trust) implemented in MATLAB (Version R2020b, The MathWorks Inc.). Data were preprocessed by slice time correction to the middle slice, movement correction and realignment to the mean image, coregistration of the functional data to individual structural scans, normalization of functional and structural images into the standard Montreal Neurological Institute (MNI) space on the basis of segmentation parameters, and spatial smoothing using a Gaussian kernel of FWHM of 8 mm. Furthermore, a 128-sec high-pass temporal filter was applied.
Statistical Data Analysis
fMRI Design Specifications
Statistical analyses of the fMRI data were conducted with SPM12. We used a general linear model (GLM) for serially autocorrelated observations (Worsley & Friston, 1995; Friston et al., 1994) and convolved regressors with the canonical hemodynamic response function. Regressors were original videos (ori), videos containing a structure modification (str), and videos containing a content modification (con), each comprising 48 trials. For str and con trials, the onsets of events were time-locked to the point in the video at which the modification occurred (time of modification). For ori trials, we calculated a hypothetical time of modification (mean of times that corresponded to points of structure and content modification in the nonmodified video) to serve as a comparable onset. These conditions were modeled as events as we were interested in the phasic effect of the prediction violation at the precise moment it occurred. To each of those regressors, we added a parametric modulator to model the repeated presentation of each video. The 24 novel videos were modeled as events as well, with onsets timed to the middle of the video. Two additional regressors modeled the 18 null events and the 56 question trials. The modeled activation of null events and questions was time-locked to their respective onsets. Null events were modeled as epochs, containing their full presentation time (7–10 sec), whereas questions were modeled as events. The six subject-specific rigid-body transformations obtained from realignment were included as regressors of no interest. Therefore, the GLM comprised 15 regressors in total.
As a first step, we calculated first-level t-contrasts for str > ori and con > ori as well as the direct contrasts str > con and con > str to analyze brain activity in response to the specific modification types. In addition, we calculated the first-level t-contrasts for each condition versus novel (nov) videos (ori > nov, str > nov, con > nov). We used this approach to demonstrate successful retrieval of encoded episodes (Jainta et al., 2022) and to validate that brain responses to episodic modifications were qualitatively different from novelty responses. A conjunction of str > ori and con > ori contrasts was calculated to detect shared effects of both modifications (Nichols, Brett, Andersson, Wager, & Poline, 2005). As an additional, more liberal approach to detect shared activation, we aggregated str and con modified (mod) videos to calculate the contrast mod > ori. Gray matter masking was applied on the first level of the analysis. For masking, we used the smoothed individual normalized gray matter image (8-mm FWHM), which was thresholded at .2 using ImCalc in SPM12 to create a binary mask. Second-level group analyses were performed by using one-sample t tests across participants. We applied a threshold of p < .001 on the whole-brain level and then used false discovery rate (FDR) correction at p < .05 on the cluster level to correct for multiple comparisons. Brain activation patterns were visualized with the software MRIcroGL (Version 1.2.20200331, McCausland Center for Brain Imaging, University of South Carolina).
To deepen our understanding of how prediction errors contribute to memory modification, we constructed a second GLM in which we split the str and con regressors into later false alarms and correct rejections in the post-fMRI memory test. The other regressors were the same as for the other GLM, but no parametric modulators were included. We contrasted false alarms with correct rejections, separately for each modification type, to investigate whether we can identify brain activation that predicts later false memories. However, this analysis did not yield any significant results.
Behavioral Data Analysis
The behavioral data analysis was conducted using RStudio (R Core Team, 2020; Version 1.3.1073).
To test our hypothesis that repeated presentations of modified videos in the fMRI session lead to a decrease in memory accuracy in the memory test in general, we considered the corrected hit rate (i.e., the discrimination index Pr, hit rate minus false alarm rate; Snodgrass & Corwin, 1988; ratings yes and rather yes were grouped as acceptance, and no and rather no as rejection). To better understand how memory for original and modified episodes was influenced in detail, we also analyzed hit rates and false alarm rates separately. Furthermore, we examined RTs in the memory test, which can serve as an indicator of how long it takes to retrieve information (correctly) from memory (Collins & Quillian, 1969). Longer RTs indicate increased difficulty of retrieval because of higher cognitive processing demands (Noppeney & Price, 2004; Larsen & Plunkett, 1987), which may also occur when competing versions of an episode are processed.
For the analysis of corrected hit rates as well as hit rates, false alarm rates (unmodified videos = targets, modified videos = distractors), and RTs for modified videos in the memory test (modifiedMT), we applied a 2 × 2 within-subject factorial design with the factors ModificationFMRI (yes, no) and VersionMT (str, con). For analyzing RTs for original videos in the memory test (originalMT), we applied a within-subject design with the factor VersionFMRI (ori, str, con). RTs were averaged over all trials of the same factorial combination. Several participants did not give any correct answers (i.e., rejection) in response to modifiedMT videos for one or more factorial combinations. For this reason, the number of datapoints included in this specific analysis was reduced to 23 per factorial combination.
We also conducted an explorative analysis on behavioral data from the fMRI session. We calculated the error rate and mean RT according to the within-subject factor VersionFMRI (ori, str, con, nov) per participant. No response was given in only 0.2% of all question trials, and these trials were not further considered in the analysis.
For the choice of statistical tests, data were inspected for normal distribution using the Shapiro Wilk Test. Furthermore, data were checked for extreme outliers as defined as values above quartile 3 + 3 × interquartile range or lower than quartile 1–3 × interquartile range. When data were normally distributed or could be transformed to fit normal distribution (RTs; logarithmic transformation) and showed no extreme outliers, we used conventional repeated-measures ANOVA (rmANOVA). When the prerequisites for parametric analysis were not met, we used a nonparametric rmANOVA based on aligned rank-transformed data (package ARTool; Wobbrock, Findlater, Gergle, & Higgins, 2011; corrected hit rates, hit rates, false alarm rates, error rates). Post hoc pair-wise comparisons were conducted with paired t tests or Wilcoxon signed-ranks tests (one-tailed when comparing ori and str and ori and con, two-tailed when comparing str and con; always two-tailed for explorative analysis of fMRI task). In addition, we used one-sample Wilcoxon signed-ranks tests to test whether corrected hit rates were significantly larger than zero.
As descriptive statistics, we report mean values and standard errors of the mean. For all behavioral analyses, we applied a significance level of α = .05. p values were adjusted according to the Bonferroni correction for multiple comparisons (Bonferroni, 1936). If the assumption of sphericity was violated as assessed by Mauchly's test of sphericity, we report Greenhouse–Geisser-corrected degrees of freedom and p values.
RESULTS
fMRI Results
Behavioral Performance during fMRI Session
We calculated a nonparametric rmANOVA on error rates for the fMRI task with the factor VersionFMRI (ori, str, con, nov). Descriptively, error rates were generally very low for all factor levels (Mori = .028 ± .007; Mstr = .036 ± .008; Mcon = .028 ± .006; Mnov = .073 ± .014) and did not differ significantly, F(3, 105) = 1.99, p = .12, ηp2 = .05. There was a significant effect of VersionFMRI on RTs, F(3, 105) = 14.32, p < .001, ηp2 = .29. Post hoc tests revealed that participants' took longer to respond after novel videos than after any other version (ori vs. nov: t(35) = −5.95, p < .001; str vs. nov: t(35) = −3.79, p = .003; con vs. nov: t(35) = −4.22, p < .001; Mori = 950.450 msec ± 26.813 msec; Mstr = 978.695 msec ± 30.695 msec; Mcon = 969.741 msec ± 26.250 msec; Mnov = 1034.822 msec ± 35.786 msec).
Neural Responses to Modified Episodic Cueing
First, we tested whether structure and content modifications elicit common brain activation patterns compared with original episodes. To this end, we calculated the conjunction of the whole-brain contrasts str > ori and con > ori, which revealed a significant activation cluster in right posterior IPS (pIPS) extending into dorsal angular gyrus (AG; Table 1). As a more liberal approach to detect common activation, we contrasted modified episodes, aggregated over both modification types, with original episodes (mod > ori). Again, common activation was found in pIPS, extending into middle IPS (mIPS) in both hemispheres. Shared activation over both modification types was also detected in right precuneus, inferior frontal sulcus (IFS), middle frontal gyrus (MFG), OTC, and left cerebellum (Table 1, Figure 4).
Whole-Brain Activation for Shared Activation of Both Episodic Modifications at FDR p < .05 (Cluster Level)
Localization . | H . | Cluster Extent . | MNI Coordinates . | t Value . | ||
---|---|---|---|---|---|---|
x . | y . | z . | ||||
(str > ori)∩(con > ori) | ||||||
pIPS extending into AG | R | 147 | 33 | −67 | 56 | 5.03 |
mod > ori | ||||||
Superior parietal lobe/pIPS | L | 152 | −27 | −61 | 50 | 5.59 |
mIPS | L | l.m. | −39 | −43 | 38 | 4.03 |
pIPS | R | 600 | 33 | −67 | 35 | 6.21 |
mIPS | R | l.m. | 45 | −37 | 50 | 4.58 |
Precuneus | R | l.m. | 6 | −67 | 41 | 4.49 |
IFS | R | 207 | 42 | 11 | 35 | 5.39 |
MFG | R | l.m. | 36 | 14 | 53 | 3.89 |
OTC | R | 119 | 54 | −52 | −10 | 5.87 |
Cerebellum | L | 80 | −6 | −82 | −37 | 5.11 |
Localization . | H . | Cluster Extent . | MNI Coordinates . | t Value . | ||
---|---|---|---|---|---|---|
x . | y . | z . | ||||
(str > ori)∩(con > ori) | ||||||
pIPS extending into AG | R | 147 | 33 | −67 | 56 | 5.03 |
mod > ori | ||||||
Superior parietal lobe/pIPS | L | 152 | −27 | −61 | 50 | 5.59 |
mIPS | L | l.m. | −39 | −43 | 38 | 4.03 |
pIPS | R | 600 | 33 | −67 | 35 | 6.21 |
mIPS | R | l.m. | 45 | −37 | 50 | 4.58 |
Precuneus | R | l.m. | 6 | −67 | 41 | 4.49 |
IFS | R | 207 | 42 | 11 | 35 | 5.39 |
MFG | R | l.m. | 36 | 14 | 53 | 3.89 |
OTC | R | 119 | 54 | −52 | −10 | 5.87 |
Cerebellum | L | 80 | −6 | −82 | −37 | 5.11 |
H = Hemisphere; L = Left; R = Right; str = structure modification; con = content modification; ori = original; mod = modification (aggregated); l.m. = local maximum.
Whole-brain activation for episodic modifications, aggregated over both modification types. FDR-corrected (p < .05) t-map for mod > ori contrast. Ori = original; mod = modification (aggregated); PCUN = precuneus.
Whole-brain activation for episodic modifications, aggregated over both modification types. FDR-corrected (p < .05) t-map for mod > ori contrast. Ori = original; mod = modification (aggregated); PCUN = precuneus.
To investigate which brain regions specifically respond to structure modifications in episodes, we inspected the contrast str > ori. Compared with episodes without modification, structurally modified episodes activated right SFS/PMd, MFG, IFS (Brodmann's area [BA] 44 and 45), supramarginal gyrus (SMG), posterior superior temporal sulcus (pSTS), IPS, and AG. In addition, we found activation in left and right precuneus (Table 2, Figure 5A).
Whole-Brain Activation for Different Episodic Modifications at FDR p < .05 (Cluster Level)
Localization . | H . | Cluster Extent . | MNI Coordinates . | t Value . | ||
---|---|---|---|---|---|---|
x . | y . | z . | ||||
str > ori | ||||||
SFS/PMd | R | 389 | 30 | 8 | 53 | 5.25 |
MFG, extending into IFS (BA 44/45) | R | l.m. | 39 | 8 | 38 | 5.08 |
SMG | R | 501 | 45 | −40 | 47 | 6.15 |
Posterior superior temporal gyrus | R | l.m. | 57 | −49 | 23 | 5.47 |
IPS | R | l.m. | 42 | −46 | 41 | 5.35 |
AG | R | l.m. | 39 | −64 | 32 | 5.29 |
Precuneus | L | 173 | −3 | −64 | 38 | 5.12 |
R | l.m. | 6 | −64 | 62 | 4.02 | |
con > ori | ||||||
pIPS | L | 361 | −27 | −61 | 50 | 7.22 |
R | 489 | 33 | −67 | 35 | 7.08 | |
IFS (BA 44) | R | 127 | 42 | 8 | 32 | 5.18 |
BA 45 | R | l.m. | 42 | 23 | 26 | 3.84 |
OTC | L | 282 | −42 | −58 | −7 | 7.76 |
Fusiform gyrus, extending into PHG | L | l.m. | −33 | −46 | −16 | 6.13 |
OTC | R | 299 | 51 | −52 | −10 | 9.07 |
Fusiform gyrus, extending into PHG | R | l.m. | 30 | −40 | −19 | 5.70 |
Cerebellum | L | 75 | −6 | −82 | −34 | 5.71 |
Localization . | H . | Cluster Extent . | MNI Coordinates . | t Value . | ||
---|---|---|---|---|---|---|
x . | y . | z . | ||||
str > ori | ||||||
SFS/PMd | R | 389 | 30 | 8 | 53 | 5.25 |
MFG, extending into IFS (BA 44/45) | R | l.m. | 39 | 8 | 38 | 5.08 |
SMG | R | 501 | 45 | −40 | 47 | 6.15 |
Posterior superior temporal gyrus | R | l.m. | 57 | −49 | 23 | 5.47 |
IPS | R | l.m. | 42 | −46 | 41 | 5.35 |
AG | R | l.m. | 39 | −64 | 32 | 5.29 |
Precuneus | L | 173 | −3 | −64 | 38 | 5.12 |
R | l.m. | 6 | −64 | 62 | 4.02 | |
con > ori | ||||||
pIPS | L | 361 | −27 | −61 | 50 | 7.22 |
R | 489 | 33 | −67 | 35 | 7.08 | |
IFS (BA 44) | R | 127 | 42 | 8 | 32 | 5.18 |
BA 45 | R | l.m. | 42 | 23 | 26 | 3.84 |
OTC | L | 282 | −42 | −58 | −7 | 7.76 |
Fusiform gyrus, extending into PHG | L | l.m. | −33 | −46 | −16 | 6.13 |
OTC | R | 299 | 51 | −52 | −10 | 9.07 |
Fusiform gyrus, extending into PHG | R | l.m. | 30 | −40 | −19 | 5.70 |
Cerebellum | L | 75 | −6 | −82 | −34 | 5.71 |
H = Hemisphere; L = Left; R = Right; str = structure modification; con = content modification; ori = original; l.m. = local maximum.
Whole-brain activation for different episodic modifications. (A) FDR-corrected (p < .05) t map for str > ori contrast. (B) FDR-corrected (p < .05) t-map for con > ori contrast. Ori = original; str = structure modification; con = content modification; (p)IPS = (posterior) intraparietal sulcus; PCUN = precuneus.
Whole-brain activation for different episodic modifications. (A) FDR-corrected (p < .05) t map for str > ori contrast. (B) FDR-corrected (p < .05) t-map for con > ori contrast. Ori = original; str = structure modification; con = content modification; (p)IPS = (posterior) intraparietal sulcus; PCUN = precuneus.
To characterize brain responses to content modification during episodic cueing, we investigated the contrast con > ori. Compared with episodes without modification, content modifications bilaterally elicited higher activity in pIPS and OTC, including FG and parahippocampal gyrus (PHG). In the right hemisphere, there was a significant activation cluster in IFS, including BA 44 and 45. In addition, we found activation in the left cerebellum (Table 2, Figure 5B).
To further verify the specificity of brain responses to both modification types, we also investigated the direct contrasts between them (str > con, con > str). In contrast to content modifications, structure modifications elicited higher activation in right precuneus, MFG, SFS/PMd, and lingual gyrus (LG). Bilaterally, we found significant activation in SMG and pSTS, extending into posterior middle temporal gyrus (pMTG) in the right hemisphere (Table 3, Figure 6A). Content compared with structure modifications triggered an elevated brain response in pIPS and OTC, including FG and PHG (Table 3, Figure 6B).
Whole-Brain Activation for Direct Contrast of Different Episodic Modifications at FDR p < .05 (Cluster Level)
Localization . | H . | Cluster extent . | MNI Coordinates . | t Value . | ||
---|---|---|---|---|---|---|
x . | y . | z . | ||||
str > con | ||||||
Precuneus | R | 76 | 9 | −49 | 62 | 5.45 |
SMG | R | 602 | 51 | −37 | 32 | 6.21 |
pSTS, extending into pMTG | R | l.m. | 45 | −34 | 2 | 4.80 |
MFG | R | 320 | 42 | 35 | 32 | 5.65 |
SFS/PMd | R | l.m. | 24 | 14 | 59 | 5.28 |
pSTS | L | 295 | −57 | −64 | 17 | 5.01 |
SMG | L | l.m. | −57 | −40 | 29 | 4.08 |
LG | R | 71 | 12 | −79 | −4 | 5.09 |
con > str | ||||||
pIPS | R | 81 | 27 | −55 | 47 | 5.00 |
L | 218 | −24 | −61 | 44 | 5.97 | |
OTC | R | 435 | 48 | −61 | −10 | 10.92 |
Fusiform gyrus | R | l.m. | 30 | −43 | −19 | 7.14 |
PHG | R | l.m. | 18 | −31 | −16 | 3.87 |
OTC | L | 349 | −42 | −61 | −7 | 8.26 |
Fusiform gyrus | L | l.m. | −33 | −49 | −16 | 5.87 |
PHG | L | l.m. | −33 | −34 | −16 | 5.09 |
Localization . | H . | Cluster extent . | MNI Coordinates . | t Value . | ||
---|---|---|---|---|---|---|
x . | y . | z . | ||||
str > con | ||||||
Precuneus | R | 76 | 9 | −49 | 62 | 5.45 |
SMG | R | 602 | 51 | −37 | 32 | 6.21 |
pSTS, extending into pMTG | R | l.m. | 45 | −34 | 2 | 4.80 |
MFG | R | 320 | 42 | 35 | 32 | 5.65 |
SFS/PMd | R | l.m. | 24 | 14 | 59 | 5.28 |
pSTS | L | 295 | −57 | −64 | 17 | 5.01 |
SMG | L | l.m. | −57 | −40 | 29 | 4.08 |
LG | R | 71 | 12 | −79 | −4 | 5.09 |
con > str | ||||||
pIPS | R | 81 | 27 | −55 | 47 | 5.00 |
L | 218 | −24 | −61 | 44 | 5.97 | |
OTC | R | 435 | 48 | −61 | −10 | 10.92 |
Fusiform gyrus | R | l.m. | 30 | −43 | −19 | 7.14 |
PHG | R | l.m. | 18 | −31 | −16 | 3.87 |
OTC | L | 349 | −42 | −61 | −7 | 8.26 |
Fusiform gyrus | L | l.m. | −33 | −49 | −16 | 5.87 |
PHG | L | l.m. | −33 | −34 | −16 | 5.09 |
H = Hemisphere; L = Left; R = Right; str = structure modification; con = content modification; ori = original; l.m. = local maximum.
Whole-brain activation for direct contrasts between episodic modifications. (A) FDR-corrected (p < .05) t-map for str > con contrast. (B) FDR-corrected (p < .05) t-map for con > str contrast. Str = structure modification; con = content modification; PCUN = precuneus.
Whole-brain activation for direct contrasts between episodic modifications. (A) FDR-corrected (p < .05) t-map for str > con contrast. (B) FDR-corrected (p < .05) t-map for con > str contrast. Str = structure modification; con = content modification; PCUN = precuneus.
Neural Effects of Episodic Reactivation
In addition to our main research question, we investigated which brain regions were activated during episodic retrieval in general. To this end, we contrasted each type of episode (ori, str, con) with novel videos (nov). For all three episode types, we found significant activation in LG (only right for con > nov), cuneus and precuneus. Original episodes and those with structure modifications additionally activated posterior cingulate cortex (PCC) and ACC. For structurally modified episodes, ACC activation extended into medial frontal gyrus and we found another significant activation cluster in right AG (Table 4, Figure 7).
Whole-Brain Activation for Episodic recall at FDR p < .05 (Cluster Level)
Localization . | H . | Cluster Extent . | MNI Coordinates . | t Value . | ||
---|---|---|---|---|---|---|
x . | y . | z . | ||||
ori > nov | ||||||
PCC | R + L | 168 | 0 | −22 | 32 | 7.73 |
ACC | L | 292 | −6 | 26 | 23 | 5.21 |
R | l.m. | 6 | 26 | 20 | 5.18 | |
LG | R | 2508 | 6 | −91 | −4 | 13.11 |
Cuneus, extending into precuneus | R | l.m. | 9 | −88 | 38 | 11.13 |
L | l.m. | −3 | −85 | 14 | 9.25 | |
str > nov | ||||||
AG | R | 121 | 48 | −55 | 53 | 5.74 |
Medial frontal gyrus | R + L | 242 | 0 | 32 | 35 | 5.63 |
ACC | L | l.m. | −3 | 32 | 26 | 4.99 |
R | l.m. | 9 | 38 | 11 | 4.42 | |
PCC | R + L | 144 | 3 | −22 | 32 | 9.02 |
LG | R | 2459 | 9 | −91 | −7 | 13.30 |
Cuneus, extending into precuneus | R | l.m. | 18 | −85 | 20 | 8.99 |
L | l.m. | −3 | −85 | 14 | 7.91 | |
con > nov | ||||||
LG | R | 1382 | 9 | −88 | −7 | 11.66 |
Cuneus | R | l.m. | 6 | −88 | 35 | 5.93 |
L | l.m. | 0 | −94 | 20 | 5.37 | |
Precuneus | R | l.m. | 15 | −64 | 32 | 5.84 |
L | l.m. | −9 | −70 | 38 | 5.55 |
Localization . | H . | Cluster Extent . | MNI Coordinates . | t Value . | ||
---|---|---|---|---|---|---|
x . | y . | z . | ||||
ori > nov | ||||||
PCC | R + L | 168 | 0 | −22 | 32 | 7.73 |
ACC | L | 292 | −6 | 26 | 23 | 5.21 |
R | l.m. | 6 | 26 | 20 | 5.18 | |
LG | R | 2508 | 6 | −91 | −4 | 13.11 |
Cuneus, extending into precuneus | R | l.m. | 9 | −88 | 38 | 11.13 |
L | l.m. | −3 | −85 | 14 | 9.25 | |
str > nov | ||||||
AG | R | 121 | 48 | −55 | 53 | 5.74 |
Medial frontal gyrus | R + L | 242 | 0 | 32 | 35 | 5.63 |
ACC | L | l.m. | −3 | 32 | 26 | 4.99 |
R | l.m. | 9 | 38 | 11 | 4.42 | |
PCC | R + L | 144 | 3 | −22 | 32 | 9.02 |
LG | R | 2459 | 9 | −91 | −7 | 13.30 |
Cuneus, extending into precuneus | R | l.m. | 18 | −85 | 20 | 8.99 |
L | l.m. | −3 | −85 | 14 | 7.91 | |
con > nov | ||||||
LG | R | 1382 | 9 | −88 | −7 | 11.66 |
Cuneus | R | l.m. | 6 | −88 | 35 | 5.93 |
L | l.m. | 0 | −94 | 20 | 5.37 | |
Precuneus | R | l.m. | 15 | −64 | 32 | 5.84 |
L | l.m. | −9 | −70 | 38 | 5.55 |
H = Hemisphere; L = Left; R = Right; str = structure modification; con = content modification; ori = original; nov = novel; l.m. = local maximum.
Whole-brain activation for videos showing original or slightly modified episodes, contrasted with novel videos. (A) FDR-corrected (p < .05) t map for ori > nov contrast. (B) FDR-corrected (p < .05) t-map for str > nov contrast. (C) FDR-corrected (p < .05) t-map for con > nov contrast. Ori = original; str = structure modification; con = content modification; nov = novel ; CUN = cuneus; PCUN = precuneus.
Whole-brain activation for videos showing original or slightly modified episodes, contrasted with novel videos. (A) FDR-corrected (p < .05) t map for ori > nov contrast. (B) FDR-corrected (p < .05) t-map for str > nov contrast. (C) FDR-corrected (p < .05) t-map for con > nov contrast. Ori = original; str = structure modification; con = content modification; nov = novel ; CUN = cuneus; PCUN = precuneus.
Post-fMRI Memory Test
Corrected Hit Rates
First, we investigated the general memory accuracy in the memory test, using the corrected hit rate. We confirmed that participants did not merely guess when rating videos as corrected hit rates for each factorial combination were significantly larger than zero (no-str: z = −4.43, p < .001; yes-str: z = −4.17, p < .001; no-con: z = −5.44, p < .001; yes-con: z = −5.30, p < .001). A nonparametric rmANOVA with the factors ModificationFMRI (yes, no) and VersionMT (str, con) revealed a significant main effect of ModificationFMRI, F(1, 35) = 10.17, p = .003, ηp2 = .23, which was driven by higher corrected hit rates for no (M = .580 ± .041) compared with yes (M = .460 ± .035), indicating a better memory performance when no modifications had been presented during the fMRI session. There was also a significant main effect of VersionMT, F(1, 35) = 109.65, p < .001, ηp2 = .76, which was explained by higher corrected hit rates values for con (M = .757 ± .038) than for str (M = .283 ± .042). There was no significant interaction of ModificationFMRI and VersionMT, F(1, 35) = 0.11, p = .75, ηp2 = .00 (Figure 8A).
Behavioral results from post-fMRI memory test. For modifiedMT and originalMT videos, participants rated whether they showed originally encoded episodes or not. OriginalMT videos were the targets whereas modifiedMT videos were distractors. (A) Corrected hit rate. Statistics: nonparametric rmANOVA with the factors ModificationFMRI (yes, no) and VersionMT (str, con), n = 36. (B) Hit rates for originalMT videos. Statistics: nonparametric rmANOVA with the factors ModificationFMRI (yes, no) and VersionMT (str, con) and Wilcoxon signed-ranks tests, n = 36. (C) False alarm rates for modifiedMT videos. Statistics: nonparametric rmANOVA with the factors ModificationFMRI (yes, no) and VersionMT (str, con), n = 36. (D) RTs for modifiedMT videos. Statistics: rmANOVA with the factors ModificationFMRI (yes, no) and VersionMT (str, con), n = 23. (E) RTs for originalMT videos. Statistics: rmANOVA with the factor VersionFMRT (ori, str, con), n = 36. Bar plots show means and standard errors. *p < .05, **p < .01, ***p < .001. Ori = original; str = structure modification; con = content modification; a versus b indicates the main effect of ModificationFMRI.
Behavioral results from post-fMRI memory test. For modifiedMT and originalMT videos, participants rated whether they showed originally encoded episodes or not. OriginalMT videos were the targets whereas modifiedMT videos were distractors. (A) Corrected hit rate. Statistics: nonparametric rmANOVA with the factors ModificationFMRI (yes, no) and VersionMT (str, con), n = 36. (B) Hit rates for originalMT videos. Statistics: nonparametric rmANOVA with the factors ModificationFMRI (yes, no) and VersionMT (str, con) and Wilcoxon signed-ranks tests, n = 36. (C) False alarm rates for modifiedMT videos. Statistics: nonparametric rmANOVA with the factors ModificationFMRI (yes, no) and VersionMT (str, con), n = 36. (D) RTs for modifiedMT videos. Statistics: rmANOVA with the factors ModificationFMRI (yes, no) and VersionMT (str, con), n = 23. (E) RTs for originalMT videos. Statistics: rmANOVA with the factor VersionFMRT (ori, str, con), n = 36. Bar plots show means and standard errors. *p < .05, **p < .01, ***p < .001. Ori = original; str = structure modification; con = content modification; a versus b indicates the main effect of ModificationFMRI.
Hit Rates
Hit rates for originalMT episodes were close to ceiling for all factorial combinations. A nonparametric rmANOVA with the factors ModificationFMRI (yes, no) and VersionMT (str, con) revealed a significant main effect of ModificationFMRI, F(1, 35) = 12.43, p = .001, ηp2 = .26, which was driven by higher hit rates for no (M = .962 ± .016) than for yes (M = .929 ± .014). Thus, participants were more prone to erroneously reject original episodes after the presentation of modified videos during the fMRI session. In addition, we found a significant main effect of VersionMT, F(1, 35) = 5.36, p = .027, ηp2 = .13, with higher hit rates for con (M = .955 ± .011) than for str (M = .936 ± .015). This indicates that participants were generally better at recognizing originally encoded episodes of which they also knew the content-modified version. The interaction of both factors was also significant, F(1, 35) = 8.18, p = .007, ηp2 = .19, and post hoc pairwise comparisons revealed that hit rates only decreased significantly after pre-experience with structure (z = −2.70, p = .015), but not content-modified episodes (z = −0.47, p = 1). Please note, however, that all differences in absolute values were quite small and thus should be interpreted with caution (Figure 8B).
False Alarm Rates
We computed a nonparametric rmANOVA with the factors ModificationFMRI (yes, no) and VersionMT (str, con) to analyze false alarm rates for modifiedMT episodes. There was a significant main effect of ModificationFMRI, F(1, 35) = 4.93, p = .033, ηp2 = .12, which was driven by higher false alarm rates for yes (M = .469 ± .039) than for no (M = .382 ± .037). Thus, participants were more prone to accept modified episode videos as originally encoded when a modified version had already been presented during the fMRI session. In addition, there was a significant main effect of VersionMT, F(1, 35) = 113.51, p < .001, ηp2 = .76, as false alarm rates were higher for str (M = .653 ± .046) than for con (M = .198 ± .035). This shows that participants generally accepted videos with modified structure much more readily than alternatives with modified content. We found a nonsignificant trend for an interaction of ModificationFMRI and VersionMT, F(1, 35) = 3.25, p = .080, ηp2 = .08. Descriptively, false alarm rates for structure modified videos were increased less by the previous experience of episodic modifications than those for content-modified videos (Figure 8C). To control for a general acceptance bias, we compared false alarm rates for novel videos and modifiedMT videos using the Wilcoxon signed-ranks test (one-tailed). False alarm rates for novel videos were at a floor level (M = .007 ± .005) and significantly lower than those for modifiedMT videos (z = −5.35, p < .001; M = .425 ± .034; Figure 8C).
RTs
A rmANOVA on RTs for modifiedMT videos with the factors ModificationFMRI (yes, no) and VersionMT (str, con) revealed a near significant effect of ModificationFMRI, F(1, 22) = 3.86, p = .062, ηp2 = .15, as participants tended to take longer to correctly reject modifiedMT episodes when the same had already been presented in the scanner (M = 782.520 msec ± 81.481 msec vs. M = 665.384 msec ± 79.401 msec). Descriptively, RTs were shorter for videos with content than with structure modification (M = 661.447 msec ± 76.010 msec vs. M = 786.458 msec ± 85.829 msec), but this difference did not reach significance either, F(1, 22) = 3.06, p = .094, ηp2 = .12. We found no significant interaction effect, F(1, 22) = 0.58, p = .45, ηp2 = .03 (Figure 8D). There was a near significant effect of VersionFMRI (ori, str, con) on RTs for originalMT videos, F(1.72, 60.09) = 3.21, p = .055, ηp2 = .08. Descriptively, RTs were longest for str (M = 923.092 msec ± 117.161 msec) compared with ori (M = 697.608 msec ± 37.210 msec) and con (M = 729.762 msec ± 62.904 msec; Figure 8E).
DISCUSSION
In this study, we investigated brain and behavioral responses to violation of episodic expectancy induced by cues with modified details in structure or content. As hypothesized, brain responses differed for these two types of episodic prediction errors, reflecting the processing of divergent object and structure information. Modified episodes were mistaken for veridical originals more often in a post-fMRI memory test when already presented during the fMRI scan, whereas correct recognition of originally encoded episodes decreased. Together, findings provide evidence that different types of mnemonic prediction errors are processed differently by the brain and may contribute to memory changes.
Neural Responses to Episodes and Episodic Modifications
The presentation of original and slightly modified videos of encoded episodes recruited several brain regions associated with episodic retrieval compared with novel videos (Jeong, Chung, & Kim, 2015; Rugg & Vilberg, 2013; Wiggs, Weisberg, & Martin, 1999). The activation patterns for episodic retrieval closely resemble the one we detected in a previous study with a similar paradigm, comprising ACC, PCC, precuneus, cuneus, and LG (Jainta et al., 2022). Thus, in parallel to replicating our previous findings, we validated that participants had successfully encoded episodes and that the presentation of videos thereof cued episodic memories. Importantly, this was true for original and for modified episodes. This was to be expected because of the subtle changes in modified episodes, which, overall, were still highly familiar to the participants.
New content and structure information of the episodic cue was expected to draw on distinct brain areas, but also to share some common activation in medial frontal cortex and the hippocampal formation. Although we could not confirm this hypothesis in this study, we found significant common activation in (right) pIPS, as revealed by the conjunction analysis and the aggregated modification contrast. Therefore, we suggest that superior parietal regions might be involved in processing of prediction errors in the context of episodic memory, potentially by guiding updating mechanisms. This interpretation fits the finding that dorsal parietal cortex plays an important role in the formation of episodic memories (Uncapher & Wagner, 2009). In addition, this area is well known to be involved in the reorientation of attention to salient and unexpected stimuli (Molenberghs, Mesulam, Peeters, & Vandenberghe, 2007; Corbetta & Shulman, 2002). It has been suggested that the superior parietal lobe, including pIPS, regulates top–down attention in memory. This is especially important when additional postretrieval processes are necessary to discriminate between what is true memorized content and what is not (Cabeza et al., 2011; Ciaramelli, Grady, & Moscovitch, 2008).
Furthermore, the more liberal approach of aggregating episodes over different types of modification revealed common activation in right IFS. This reflects our previous finding that activity in ventrolateral pFC increases for inconsistent or highly informative detail in observed actions (El-Sourani et al., 2019; Hrkać, Wurm, Kühn, & Schubotz, 2015; Wurm & Schubotz, 2012).
Structure and content modifications each recruited a set of brain regions unique to the modification type. Activation patterns in contrast to original episodes closely resembled those for the direct contrasts between structure and content modifications, indicating high specificity of brain responses for each modification type.
On the one hand, we had expected that structure modifications specifically lead to activation in brain regions involved in the temporal organization of episodes. We found that structure modifications co-activated right PMd/SFS, SMG, pSTS, and precuneus, suggesting this network contributes to the updating of predictive models because of unexpected new structure in episodes. This interpretation is consistent with previous reports about the functional characteristics of said areas. Accordingly, activity of a region comprising PMd/SFS is related to linking successive action steps (Pomp et al., 2021; Hrkać et al., 2014; Schubotz et al., 2012; Stadler et al., 2011) and could contribute to updating the current event or action model with respect to each next segment (Pomp et al., 2021; Schubotz et al., 2012; Tamber-Rosenau et al., 2011; Kurby & Zacks, 2008). SMG and precuneus have been demonstrated to be important for the sequential organization of memories (Foudil, Kwok, & Macaluso, 2020; Guidali, Pisoni, Bolognini, & Papagno, 2019) and involved in sequential learning (Burke, Bramley, Gonzalez, & McKeefry, 2013; Oishi et al., 2005). In line with this, we recently found that SMG is sensitive for perceived break points in actions (Pomp et al., 2021). Activity in (right) pSTS is characteristic for the processing of biological motion (Gilaie-Dotan, Kanai, Bahrami, Rees, & Saygin, 2013; Grossman, Battelli, & Pascual-Leone, 2005) and, in this context, action adaptation (Thurman, van Boxtel, Monti, Chiang, & Lu, 2016). pSTS activation for actions has been found to be goal-sensitive, responding more strongly when expected spatial transport targets are not met (Shultz, Lee, Pelphrey, & McCarthy, 2011).
On the other hand, we found content modifications to specifically recruit pIPS and OTC, including FG, which were hypothesized on the basis of their role for processing of object properties in the context of actions (El-Sourani et al., 2019; Lingnau & Downing, 2015; Wiggett & Downing, 2011; Reber et al., 2005; Grill-Spector, Kourtzi, & Kanwisher, 2001). More specifically, pIPS encodes basic visual features of graspable objects (Mruczek, von Loga, & Kastner, 2013; Creem-Regehr, 2009), reflecting the interaction with toy objects in our paradigm. In addition, content modifications elicited activity in the hippocampal formation (PHG), which likely represents ongoing learning because of the detected mismatch. In general, hippocampus and PHG are important in learning contexts (Davachi & Wagner, 2002; Köhler, Crane, & Milner, 2002; O'Reilly & Rudy, 2000; Aguirre, Detre, Alsop, & D'Esposito, 1996) and there is evidence that the PHG is involved in processing of competing memories (Kuhl, Bainbridge, & Chun, 2012). Moreover, the hippocampal formation is believed to generate mismatch signals when predictions do not fit perceptual inputs (Long et al., 2016; Duncan, Curtis, & Davachi, 2009; Kumaran & Maguire, 2007). Because our post-fMRI memory test data imply that content changes were more salient than structural changes, one could speculate that the discrepancy between what was predicted and what was perceived in case of content modifications was strong enough to be reflected in the activation of the hippocampal formation. Interestingly, the overall activation pattern we found for content modifications closely resembles the one Gläscher, Daw, Dayan, and O'Doherty (2010) identified for what they call state prediction errors, which is characterized by a mismatch of the expected and current state.
Taken together, we found that structure and content modifications activated distinct networks, each specifically representing the processing of the type of unexpected new information. We therefore achieved the main aim of this study, that is, characterize brain responses to different types of prediction errors.
Influence of Episodic Modifications on Post-fMRI Memory Performance
As expected, those original episodes that had been presented in a modified version during the fMRI were recalled less reliably, as reflected by a lower corrected hit rate. More specifically, previously modified videos were later more often mistaken for original ones, as evidenced by significantly increased false alarm rates. Others have reported that mnemonic prediction errors lead to an intrusion of new information into an established memory repertoire (Sinclair & Barense, 2018; Long et al., 2016). Also, in our study, the recognition of original videos was impaired after encounters of alternative versions. Similarly, Kim et al. reported that prediction violations led to decreased recognition of original memory content (Kim et al., 2014). In particular, prior presentation of structurally modified videos led to a decrease in the hit rate in the post-fMRI memory test. This might be taken as a first hint that different types of episodic modifications could influence memory traces differently, which would fit the specific brain responses we detected for structure and content modifications. Interestingly, participants tended to take longer to correctly classify a video if they had already seen the same video in a modified version during fMRI. Longer response times in cued-memory paradigms are interpreted as indicative of increased difficulty of retrieval because of higher cognitive processing demands (Noppeney & Price, 2004; Larsen & Plunkett, 1987). Thus, it is likely that it became more difficult to differentiate between alternative competing versions of episodes when versions diverging from the original experience had been already encountered in the fMRI session. However, effects concerning RTs need to be interpreted with caution because they only approached significance. Taken together, the behavioral findings suggest that structure and content modifications during cueing of episodic retrieval influenced subsequent memory for these episodes. Our findings corroborate the observation that mnemonic prediction errors can trigger episodic memory modification (Sinclair & Barense, 2019).
What remains unclear is how exactly memory traces were influenced by our intervention. For example, it has been discussed that memory modification can result from an interference of old and new memory traces (Sinclair & Barense, 2018, 2019; Klingmüller, Caplan, & Sommer, 2017; Sederberg, Gershman, Polyn, & Norman, 2011) or from source confusion (Hekkanen & McEvoy, 2002), which could both explain our findings. Then again, participants remembered correctly that novel videos had not been part of the original episode repertoire although novels had been repeatedly presented during the fMRI experiment as well. This speaks against source confusion in its simplest form as an explanation of the results of our post-fMRI memory test.
Another interesting behavioral finding was that participants generally had a strong tendency to accept structurally modified versions as originals in the memory test. Although both types of modifications resulted from a single change in the story, it is likely that structural modifications were generally less salient than content modifications. This would be matched by the fact that after the fMRI session, nearly all participants (86%) reported noticing at least one object swap, whereas only half of them had noticed a change in the sequence of action steps (53%). Recently, it was reported that memory performance following prediction errors differed depending on whether changes were detected (and remembered) by participants or not (Wahlheim & Zacks, 2019). Whereas undetected changes led to reduced memory performance, detected changes had the opposite effect. Depending on contextual factors, prediction errors can even improve subsequent memory (Greve, Cooper, Kaula, Anderson, & Henson, 2017; Smith, Hasinski, & Sederberg, 2013). Although our behavioral results suggest that structural changes were rarely noticed, corresponding to a reduced memory performance, more frequent detection in the case of content modification did not lead to enhanced, but on the contrary, also to decreased memory performance. Thus, the final impact of a prediction violation on memory appears to be multifactorially determined.
Limitations and Implications for Future Research
One factor that may limit the generalizability of our findings is that, for practical reasons, only women participated in the study. However, because the processing of episodic memory in the brain seems to be broadly similar between women and men (Nyberg, Habib, & Herlitz, 2000), we are confident that our findings are applicable to a more general population. In the future, our paradigm could be adapted to circumvent such practical limitations, for example, by applying virtual reality techniques so that encoding could be detached from the true physical appearance of participants' hands.
Second, we used new content and structure information, which contrasted details of the encoded episodes to elicit prediction errors. Although these interventions can also be interpreted as contextual and associative novelty, respectively, the unexpected new input within the familiar context will give rise to mnemonic prediction errors according to the predictive coding framework (see the work of Reichardt et al., 2020, for a review). In addition, we could show that episodes with structure and content modifications, in contrast to completely novel videos, recruited different brain regions associated with episodic retrieval (Jeong et al., 2015; Rugg & Vilberg, 2013; Wiggs et al., 1999). Still, it would be interesting to find a way to keep the novelty constant and have participants make active predictions that are then either violated or not.
Our findings from the post-fMRI memory test revealed that structure modifications were likely less salient and harder to detect than content modifications. Thus, we cannot exclude that the neural differences we detected were confounded by differences in prediction error strength. However, brain responses to structure modifications were highly specific and located in hypothesized areas. Moreover, structure modifications elicited equally strong activation as content modifications. It is thus highly unlikely that those differences simply arose because of quantitative difference between modification types. For these reasons, we are confident that our findings indeed represent differential neural processing because of different types of episodic information.
As a caveat, we wish to point out that, in normal life, episodic memories are not trained or repeated in the strict sense as they were in our paradigm. Although experimental procedures must be rigorous to be able to test hypotheses, the rigor takes away from the applicability of the research.
Last, our trial-wise analysis of later false alarms and correct rejections in the memory test did not reveal significant brain activation predicting false memories. Our aim for future investigations is to optimize our paradigm to further analyze how later true and false memories are encoded by the brain.
Conclusion
When recalling episodes, our memory can change, for instance because of prediction errors in the reactivation process. Our results suggest that structural and content prediction errors in episode retrieval differ in their neural processing. The tendency to misclassify modified episodes as originally experienced episodes increased after experiencing repeated structural and content prediction errors. Accordingly, different types of prediction errors can confuse episodic memory and possibly lead to the emergence of alternative versions of the same memory trace. Our results may provide a fruitful starting point for further research on the mutability of episodic memories.
Acknowledgments
The authors thank Monika Mertens, Lena Puder, Simon Wieczorek, Jamuna Halscheid, Leandra Feldhusen, and Anne Glombitza for their help during data collection. Furthermore, we thank Annika Garlichs, Helena Sydlik, and Yuyi Xu for their assistance during the creation of stimulus material and Christin Schwarzer for training new student assistants. Last, we thank Jennifer Pomp, Lena Schliephake, Falko Mecklenbrauck, and Nina Heins for advice regarding data analysis and the members of research unit FOR 2812 for valuable discussions.
Reprint requests should be sent to Sophie Siestrup, University of Münster, Fliednerstraße 21, 48149 Münster, or via e-mail: [email protected].
Data Availability Statement
All data reported here is publicly available at https://osf.io/m7dcu/?view_only=575d6ed3fbf544ada3bcb0519c86f94b.
Author Contributions
Sophie Siestrup: Formal analysis; Investigation; Methodology; Visualization; Writing— Original draft; Writing—Review & editing. Benjamin Jainta: Investigation; Methodology; Writing—Review & editing. Nadiya El-Sourani: Methodology; Writing—Review & editing. Ima Trempler: Formal analysis; Methodology; Writing—Review & editing. Moritz F. Wurm: Writing—Review & editing. Oliver T. Wolf: Writing—Review & editing. Sen Cheng: Conceptualization; Writing—Review & editing. Ricarda I. Schubotz: Conceptualization; Funding acquisition; Methodology; Resources; Supervision; Writing—Original draft; Writing—Review & editing.
Funding Information
This work was funded by the German Research Foundation (Deutsche Forschungsgemeinschaft) – project numbers 419037023, 419039274, 419037518. The funders had no role in study design, data collection, analysis and interpretation, decision to publish, or writing of the report.
Diversity in Citation Practices
Retrospective analysis of the citations in every article published in this journal from 2010 to 2021 reveals a persistent pattern of gender imbalance: Although the proportions of authorship teams (categorized by estimated gender identification of first author/last author) publishing in the Journal of Cognitive Neuroscience (JoCN) during this period were M(an)/M = .407, W(oman)/M = .32, M/W = .115, and W/W = .159, the comparable proportions for the articles that these authorship teams cited were M/M = .549, W/M = .257, M/W = .109, and W/W = .085 (Postle and Fulvio, JoCN, 34:1, pp. 1–3). Consequently, JoCN encourages all authors to consider gender balance explicitly when selecting which articles to cite and gives them the opportunity to report their article's gender citation balance.