Motion information has been argued to be central to the subjective segmentation of observed actions. Concerning object-directed actions, object-associated action information might as well inform efficient action segmentation and prediction. The present study compared the segmentation and neural processing of object manipulations and equivalent dough ball manipulations to elucidate the effect of object–action associations. Behavioral data corroborated that objective relational changes in the form of (un-)touchings of objects, hand, and ground represent meaningful anchor points in subjective action segmentation rendering them objective marks of meaningful event boundaries. As expected, segmentation behavior became even more systematic for the weakly informative dough. fMRI data were modeled by critical subjective, and computer-vision-derived objective event boundaries. Whole-brain as well as planned ROI analyses showed that object information had significant effects on how the brain processes these boundaries. This was especially pronounced at untouchings, that is, events that announced the beginning of the upcoming action and might be the point where competing predictions are aligned with perceptual input to update the current action model. As expected, weak object–action associations at untouching events were accompanied by increased biological motion processing, whereas strong object–action associations came with an increased contextual associative information processing, as indicated by increased parahippocampal activity. Interestingly, anterior inferior parietal lobule activity increased for weak object–action associations at untouching events, presumably because of an unrestricted number of candidate actions for dough manipulation. Our findings offer new insights into the significance of objects for the segmentation of action.

Prediction errors (PEs) function as learning signals. It is yet unclear how varying compared to repetitive PEs affect episodic memory in brain and behavior. The current study investigated cerebral and behavioral effects of experiencing either multiple alternative versions (“varying”) or one single alternative version (“repetitive”) of a previously encoded episode. Participants encoded a set of episodes (“originals”) by watching videos showing toy stories. During scanning, participants either experienced originals, one single, or multiple alternative versions of the previously encoded episodes. Participants' memory performance was tested through recall of original objects. Varying and repetitive PEs revealed typical brain responses to the detection of mismatching information including inferior frontal and posterior parietal regions, as well as hippocampus, which is further linked to memory reactivation, and the amygdala, known for modulating memory consolidation. Furthermore, experiencing varying and repetitive PEs triggered distinct brain areas as revealed by direct contrast. Among others, experiencing varying versions triggered activity in the caudate, a region that has been associated with PEs. In contrast, repetitive PEs activated brain areas that resembled more those for retrieval of originally encoded episodes. Thus, ACC and posterior cingulate cortex activation seemed to serve both reactivating old and integrating new but similar information in episodic memory. Consistent with neural findings, participants recalled original objects less accurately when only presented with the same, but not varying, PE during fMRI. The current findings suggest that repeated PEs interact more strongly with a recalled original episodic memory than varying PEs.

Episodic memories can be modified, a process that is potentially driven by mnemonic prediction errors. In the present study, we used modified cues to induce prediction errors of different episodic relevance. Participants encoded episodes in the form of short toy stories and then returned for an fMRI session on the subsequent day. Here, participants were presented either original episodes or slightly modified versions thereof. Modifications consisted of replacing a single object within the episode and either challenged the gist of an episode (gist modifications) or left it intact (surface modifications). On the next day, participants completed a post-fMRI memory test that probed memories for originally encoded episodes. Both types of modifications triggered brain activation in regions we previously found to be involved in the processing of content-based mnemonic prediction errors (i.e., the exchange of an object). Specifically, these were ventrolateral pFC, intraparietal cortex, and lateral occipitotemporal cortex. In addition, gist modifications triggered pronounced brain responses, whereas those for surface modification were only significant in the right inferior frontal sulcus. Processing of gist modifications also involved the posterior temporal cortex and the precuneus. Interestingly, our findings confirmed the posterior hippocampal role of detail processing in episodic memory, as evidenced by increased posterior hippocampal activity for surface modifications compared with gist modifications. In the post-fMRI memory test, previous experience with surface modified, but not gist-modified episodes, increased erroneous acceptance of the same modified versions as originally encoded. Whereas surface-level prediction errors might increase uncertainty and facilitate confusion of alternative episode representations, gist-level prediction errors seem to trigger the clear distinction of independent episodes.

How susceptible a memory is to later modification might depend on how stable the episode has been encoded. This stability was proposed to increase when retrieving information more (vs. less) often and in a spaced (vs. massed) practice. Using fMRI, we examined the effects of these different pre-fMRI retrieval protocols on the subsequent propensity to learn from episodic prediction errors. After encoding a set of different action stories, participants came back for two pre-fMRI retrieval sessions in which they encountered original episodes either two or eight times in either a spaced or a massed retrieval protocol. One week later, we cued episodic retrieval during the fMRI session by using original or modified videos of encoded action stories. Recurrent experience of modified episodes was associated with increasing activity in the episodic memory network including hippocampal and cortical areas, when leading to false memories in a post-fMRI memory test. While this observation clearly demonstrated learning from episodic prediction errors, we found no evidence for a modulatory effect of the different retrieval protocols. As expected, the benefit of retrieving an episode more often was reflected in better memory for originally encoded episodes. In addition, frontal activity increased for episodic prediction errors when episodes had been less frequently retrieved pre-fMRI. A history of spaced versus massed retrieval was associated with increased activation throughout the episodic memory network, with no significant effect on behavioral performance. Our findings show that episodic prediction errors led to false memories. The history of different retrieval protocols was reflected in memory performance and brain responses to episodic prediction errors, but did not interact with the brain's episodic learning response.

Episodic memories are not static but can change on the basis of new experiences, potentially allowing us to make valid predictions in the face of an ever-changing environment. Recent research has identified prediction errors during memory retrieval as a possible trigger for such changes. In this study, we used modified episodic cues to investigate whether different types of mnemonic prediction errors modulate brain activity and subsequent memory performance. Participants encoded episodes that consisted of short toy stories. During a subsequent fMRI session, participants were presented videos showing the original episodes, or slightly modified versions thereof. In modified videos, either the order of two subsequent action steps was changed or an object was exchanged for another. Content modifications recruited parietal, temporo-occipital, and parahippocampal areas reflecting the processing of the new object information. In contrast, structure modifications elicited activation in right dorsal premotor, posterior temporal, and parietal areas, reflecting the processing of new sequence information. In a post-fMRI memory test, the participants' tendency to accept modified episodes as originally encoded increased significantly when they had been presented modified versions already during the fMRI session. After experiencing modifications, especially those of the episodes' structure, the recognition of originally encoded episodes was impaired as well. Our study sheds light onto the neural processing of different types of episodic prediction errors and their influence on subsequent memory recall.

The processing of congruent stimuli, such as an object or action in its typical location, is usually associated with reduced neural activity, probably due to facilitated recognition. However, in some situations, congruency increases neural activity—for example, when objects next to observed actions are likely versus unlikely to be involved in forthcoming action steps. Here, we investigated using fMRI whether the processing of contextual cues during action perception is driven by their (in)congruency and, thus, informative value to make sense of an observed scene. Specifically, we tested whether both highly congruent contextual objects (COs), which strongly indicate a future action step, and highly incongruent COs, which require updating predictions about possible forthcoming action steps, provide more anticipatory information about the action course than moderately congruent COs. In line with our hypothesis that especially the inferior frontal gyrus (IFG) subserves the integration of the additional information into the predictive model of the action, we found highly congruent and incongruent COs to increase bilateral activity in action observation nodes, that is, the IFG, the occipitotemporal cortex, and the intraparietal sulcus. Intriguingly, BA 47 was significantly stronger engaged for incongruent COs reflecting the updating of prediction in response to conflicting information. Our findings imply that the IFG reflects the informative impact of COs on observed actions by using contextual information to supply and update the currently operating predictive model. In the case of an incongruent CO, this model has to be reconsidered and extended toward a new overarching action goal.

Surprising events may be relevant or irrelevant for behavior, requiring either flexible adjustment or stabilization of our model of the world and according response strategies. Cognitive flexibility and stability in response to environmental demands have been described as separable cognitive states, associated with activity of striatal and lateral prefrontal regions, respectively. It so far remains unclear, however, whether these two states act in an antagonistic fashion and which neural mechanisms mediate the selection of respective responses, on the one hand, and a transition between these states, on the other. In this study, we tested whether the functional dichotomy between striatal and prefrontal activity applies for the separate functions of updating (in response to changes in the environment, i.e., switches) and shielding (in response to chance occurrences of events violating expectations, i.e., drifts) of current predictions. We measured brain activity using fMRI while 20 healthy participants performed a task that required to serially predict upcoming items. Switches between predictable sequences had to be indicated via button press while sequence omissions (drifts) had to be ignored. We further varied the probability of switches and drifts to assess the neural network supporting the transition between flexible and stable cognitive states as a function of recent performance history in response to environmental demands. Flexible switching between models was associated with activation in medial pFC (BA 9 and BA 10), whereas stable maintenance of the internal model corresponded to activation in the lateral pFC (BA 6 and inferior frontal gyrus). Our findings extend previous studies on the interplay of flexibility and stability, suggesting that different prefrontal regions are activated by different types of prediction errors, dependent on their behavioral requirements. Furthermore, we found that striatal activation in response to switches and drifts was modulated by participants' successful behavior toward these events, suggesting the striatum to be responsible for response selections following unpredicted stimuli. Finally, we observed that the dopaminergic midbrain modulates the transition between different cognitive states, thresholded by participants' individual performance history in response to temporal environmental demands.

Because everyday actions are statistically structured, knowing which action a person has just completed allows predicting the most likely next action step. Taking even more than the preceding action into account improves this predictability but also causes higher processing costs. Using fMRI, we investigated whether observers exploit second-order statistical regularities preferentially if information on possible upcoming actions provided by first-order regularities is insufficient. We hypothesized that anterior pFC balances whether or not second-order information should be exploited. Participants watched videos of actions that were structured by first- and second-order conditional probabilities. Information provided by the first and by the second order was manipulated independently. BOLD activity in the action observation network was more attenuated the more information on upcoming actions was provided by first-order structure, reflecting expectation suppression for more predictable actions. Activation in posterior parietal sites decreased further with second-order information but increased in temporal areas. As expected, second-order information was integrated more when less first-order information was provided, and this interaction was mediated by anterior pFC (BA 10). Observers spontaneously used both the present and the preceding action to predict the upcoming action, and integration of the preceding action was enhanced when the present action was uninformative.

To recognize an action, an observer exploits information about the applied manipulation, the involved objects, and the context where the action occurs. Context, object, and manipulation information are hence expected to be tightly coupled in a triadic relationship (the COM triad hereafter). The current fMRI study investigated the hemodynamic signatures of reciprocal modulation in the COM triad. Participants watched short video clips of pantomime actions, that is, actions performed with inappropriate objects, taking place at compatible or incompatible contexts. The usage of pantomime actions enabled the disentanglement of the neural substrates of context–manipulation (CM) and context–object (CO) associations. There were trials in which (1) both manipulation and objects, (2) only manipulation, (3) only objects, or (4) neither manipulation nor objects were compatible with the context. CM compatibility effects were found in an action-related network comprising ventral premotor cortex, SMA, left anterior intraparietal sulcus, and bilateral occipito-temporal cortex. Conversely, CO compatibility effects were found bilaterally in lateral occipital complex. These effects interacted in subregions of the lateral occipital complex. An overlap of CM and CO effects was observed in the occipito-temporal cortex and the dorsal attention network, that is, superior frontal sulcus/dorsal premotor cortex and superior parietal lobe. Results indicate that contextual information is integrated into the analysis of actions. Manipulation and object information is linked by contextual associations as a function of co-occurrence in specific contexts. Activation of either CM or CO associations shifts attention to either action- or object-related relevant information.

When we observe an action, we know almost immediately what goal is pursued by the actor. Strikingly, this applies also to pretend action (pantomime), which provides relevant information about the manipulation itself but not about the manipulated objects. The present fMRI study addressed the issue of goal inference from pretend action as compared with real action. We found differences as well as commonalities for the brain correlates of inferring goals from both types of action. They differed with regard to the weights of the underlying action observation network, indicating the exploitation of object information in the case of real actions and manipulation information in the case of pretense. However, goal inferences from manipulation information resulted in a common network for both real and pretend action. Interestingly, this latter network also comprised areas that are not identified by action observation and that might be due to the processing of scene gist and to the evaluation of fit of putative action goals. These findings suggest that observation of pretense emphasizes the requirement to internally simulate the observed act but rule out fundamental differences of how observers cope with real and pretend action.

Setting perceptual expectations can be based on different sources of information that determine which functional networks will be involved in implementing preparatory top–down influences and dealing with situations in which expectations are violated. The goal of the present study was to investigate and directly compare brain activations triggered by violating expectations within two different task contexts. In the serial prediction task, participants monitored ordered perceptual sequences for predefined sequential deviants. In contrast, the target detection task entailed a presentation of stimuli which had to be monitored for predefined nonsequential deviants. Detection of sequential deviants triggered an increase of activity in premotor and cerebellar components of the “standard” sequencing network and activations in additional frontal areas initially not involved in sequencing. This pattern of activity reflects the detection of a mismatch between the expected and presented stimuli, updating of the underlying sequence representation (i.e., forward model), and elaboration of the violation. In contrast, target detection elicited activations in posterior temporal and parietal areas, reflecting an increase in perceptual processing evoked by the nonsequential deviant. The obtained results suggest that distinct functional networks involved in detecting deviants in different contexts reflect the origin and the nature of expectations being violated.

A considerable part of our lives is spent engaging in the entertaining worlds of fiction that are accessible through media such as books and television. Little is known, however, about how we are able to readily understand that fictional events are distinct from those occurring within our real world. The present functional imaging study explored the brain correlates underlying such abilities by having participants make judgments about the possibility of different scenarios involving either real or fictional characters being true, given the reality of our world. The processing of real and fictional scenarios activated a common set of regions including medial-temporal lobe structures. When the scenarios involved real people, brain regions associated with episodic memory retrieval and self-referential thinking, the anterior prefrontal cortex and the precuneus/posterior cingulate, were more active. In contrast, areas along the left lateral inferior frontal gyrus, associated with semantic memory retrieval, were implicated for scenarios with fictional characters. This implies that there is a fine distinction in the manner in which conceptual information concerning real persons in contrast to fictional characters is represented. In general terms, the findings suggest that fiction relative to reality tends to be represented in more factual terms, whereas our representations of reality relative to fiction are colored by personal subjectivity. What modulates our understanding of the relative difference between reality and fiction seems to be whether such character-type information is coded in self-relevant terms or not.

When ranking two alternatives by some criteria and only one of the alternatives is recognized, participants overwhelmingly adopt the strategy, termed the recognition heuristic (RH), of choosing the recognized alternative. Understanding the neural correlates underlying decisions that follow the RH could help determine whether people make judgments about the RH's applicability or simply choose the recognized alternative. We measured brain activity by using functional magnetic resonance imaging while participants indicated which of two cities they thought was larger (Experiment 1) or which city they recognized (Experiment 2). In Experiment 1, increased activation was observed within the anterior frontomedian cortex (aFMC), precuneus, and retrosplenial cortex when participants followed the RH compared to when they did not. Experiment 2 revealed that RH decisional processes cannot be reduced to recognition memory processes. As the aFMC has previously been associated with self-referential judgments, we conclude that RH decisional processes involve an assessment about the applicability of the RH.