While recounting an experience, one can employ multiple strategies to transition from one part to the next. For instance, if the event was learned out of linear order, one can recall events according to the time they were learned (temporal), similar events (semantic), events occurring nearby in time (chronological), or events produced by the current event (causal). To disentangle the importance of these factors, we had participants watch the nonlinear narrative, Memento, under different task instructions and presentation orders. For each scene of the film, we also separately computed semantic and causal networks. We then contrasted the evidence for temporal, semantic, chronological, or causal strategies during recall. Critically, there was stronger evidence for the causal and chronological strategies than semantic or temporal strategies. Moreover, the causal and chronological strategies outperformed the temporal one even when we asked participants to recall the film in the presented order, underscoring the fundamental nature of causal structure in scaffolding understanding and organizing recall. Nevertheless, time still marginally predicted recall transitions, suggesting it still operates as a weak signal in the presence of more salient forms of structure. In addition, semantic and causal network properties predicted scene memorability, including a stronger role for incoming causes to an event than its outgoing effects. In summary, these findings highlight the importance of accounting for complex, causal networks in knowledge building and memory.

Our continuous experience is spontaneously segmented by the brain into discrete events. However, the beginning of a new event (an event boundary) is not always sharply identifiable: Phenomenologically, event boundaries vary in salience. How are the response profiles of cortical areas at event boundaries modulated by boundary strength during complex, naturalistic movie-viewing? Do cortical responses scale in a graded manner with boundary strength, or do they merely detect boundaries in a binary fashion? We measured “cortical boundary shifts” as transient changes in multivoxel patterns at event boundaries with different strengths (weak, moderate, and strong), determined by across-participant agreement. Cortical regions with different processing timescales were examined. In auditory areas, which have short timescales, cortical boundary shifts exhibited a clearly graded profile in both group-level and individual-level analyses. In cortical areas with long timescales, including the default mode network, boundary strength modulated pattern shift magnitude at the individual participant level. We also observed a positive relationship between boundary strength and the extent of temporal alignment of boundary shifts across different levels of the cortical hierarchy. In addition, hippocampal activity was highest at event boundaries for which cortical boundary shifts were most aligned across hierarchical levels. Overall, we found that event boundary strength modulated cortical pattern shifts strongly in sensory areas and more weakly in higher-level areas and that stronger boundaries were associated with greater alignment of these shifts across the cortical hierarchy.

Recent fMRI studies of event segmentation have found that default mode regions represent high-level event structure during movie watching. In these regions, neural patterns are relatively stable during events and shift at event boundaries. Music, like narratives, contains hierarchical event structure (e.g., sections are composed of phrases). Here, we tested the hypothesis that brain activity patterns in default mode regions reflect the high-level event structure of music. We used fMRI to record brain activity from 25 participants (male and female) as they listened to a continuous playlist of 16 musical excerpts and additionally collected annotations for these excerpts by asking a separate group of participants to mark when meaningful changes occurred in each one. We then identified temporal boundaries between stable patterns of brain activity using a hidden Markov model and compared the location of the model boundaries to the location of the human annotations. We identified multiple brain regions with significant matches to the observer-identified boundaries, including auditory cortex, medial prefrontal cortex, parietal cortex, and angular gyrus. From these results, we conclude that both higher-order and sensory areas contain information relating to the high-level event structure of music. Moreover, the higher-order areas in this study overlap with areas found in previous studies of event perception in movies and audio narratives, including regions in the default mode network.

The posterior medial network is at the apex of a temporal integration hierarchy in the brain, integrating information over many seconds of viewing intact, but not scrambled, movies. This has been interpreted as an effect of temporal structure. Such structure in movies depends on preexisting event schemas, but temporal structure can also arise de novo from learning. Here, we examined the relative role of schema-consistent temporal structure and arbitrary but consistent temporal structure on the human posterior medial network. We tested whether, with repeated viewing, the network becomes engaged by scrambled movies with temporal structure. Replicating prior studies, activity in posterior medial regions was immediately locked to stimulus structure upon exposure to intact, but not scrambled, movies. However, for temporally structured scrambled movies, functional coupling within the network increased across stimulus repetitions, rising to the level of intact movies. Thus, temporal structure is a key determinant of network dynamics and function in the posterior medial network.