Abstract
Recent studies have shown that functional connectivity can be decomposed into its exact frame-wise contributions, revealing short-lived, infrequent, and high-amplitude time points referred to as “events.” Events contribute disproportionately to the time-averaged connectivity pattern, improve identifiability and brain-behavior associations, and differences in their expression have been linked to endogenous hormonal fluctuations and autism. Here, we explore the characteristics of events while subjects watch movies. Using two independently-acquired imaging datasets in which participants passively watched movies, we find that events synchronize across individuals and based on the level of synchronization, can be categorized into three distinct classes: those that synchronize at the boundaries between movies, those that synchronize during movies, and those that do not synchronize at all. We find that boundary events, compared to the other categories, exhibit greater amplitude, distinct co-fluctuation patterns, and temporal propagation. We show that underlying boundary events is a specific mode of co-fluctuation involving the activation of control and salience systems alongside the deactivation of visual systems. Events that synchronize during the movie, on the other hand, display a pattern of co-fluctuation that is time-locked to the movie stimulus. Finally, we found that subjects’ time-varying brain networks are most similar to one another during these synchronous events.
1 Introduction
The human brain is fundamentally a complex network comprising anatomically connected neural elements (Bassett & Sporns, 2017; Bullmore & Sporns, 2009; Park & Friston, 2013). This physical network constrains dynamical interactions between brain regions, inducing statistical dependencies in the activity of distant brain regions, that is, functional connectivity (FC) (Friston, 1994; Horwitz, 2003; Reid et al., 2019). A growing number of studies have focused on characterizing the architectural features of FC (Power et al., 2011, 2013) and linking inter-individual differences in these features to cognition (Avery et al., 2020; King et al., 2021; Rosenberg et al., 2016), disease (Fornito et al., 2015; Tian et al., 2019), and development (Chan et al., 2014; Gu et al., 2015).
Recent methodological advances have made it possible to precisely decompose FC into its framewise contributions (Esfahlani et al., 2020; Faskowitz et al., 2020). This “edge-centric” approach yields time-varying estimates of the co-fluctuation magnitude and valence for every pair of brain regions (edge). Previous studies have shown that, collectively, edges exhibit bursty behavior, such that long periods of quiescence are punctuated by brief, high-amplitude “events” in which many edges simultaneously and strongly co-fluctuate with one another (R.F. Betzel et al., 2022; R. Betzel et al., 2022; Cifre et al., 2020; Esfahlani et al., 2020; Greenwell et al., 2021; Liu & Duyn, 2013; Petridou et al., 2013; Tagliazucchi et al., 2012). The whole-brain co-fluctuation patterns expressed during events are closely related to the static (time-averaged) FC (Esfahlani et al., 2020), improve subject identification and brain-behavior correlations (Esfahlani et al., 2020), are individualized (R. F. Betzel et al., 2022), can be linked to endogenous hormone fluctuations (Greenwell et al., 2021), clinical status (Esfahlani et al., 2021), and memory processes (Cooper et al., 2021), and are shaped by the underlying anatomical connectivity (Pope et al., 2021).
However, the principles that determine the timing of events are undisclosed. The first “edge-centric” study showed that the whole-brain co-fluctuation amplitude is correlated during movie-watching but not at rest (Esfahlani et al., 2020), suggesting that audiovisual stimuli may induce correlations in the timing of events. However, the spatiotemporal structure of events in naturalistic stimuli was not investigated further.
Here, we characterize this synchronization in the timing of events in greater detail with the aim of contributing a deeper understanding of the potential drivers of events. As in Esfahlani et al. (2020), we find that the timing of events synchronizes across participants watching the same movie. We replicate this finding using two independently acquired, -processed, and -parcellated datasets, representing eight separate movies (each movie consisting of a small number of movie scenes, or trailers). We analyze events further, grouping them into three distinct categories: events that synchronize at the boundaries between movies, those that synchronize within the body of a movie, and those that are asynchronous. We focus further on boundary events, showing that they exhibit distinct overall co-fluctuation amplitudes, co-fluctuation pattern, and temporal structure. We show that boundary events are also underpinned by a distinct pattern of activity that involves visual processing, attentional, and control systems. Events that synchronize during movies, however, are characterized by co-fluctuation patterns that are time-locked to the movie stimulus. Additionally, we found that subjects’ time-varying brain networks are most similar to one another during these synchronous events.
Taken together, these results suggest a potential driver of one class of events, and offer a more in-depth analysis of the others. Furthermore, our results suggest that the synchronization of events is more broadly related to the phenomenon of synchronization in subjects’ brains at two different scales: the whole-brain co-fluctuation amplitude, and the edge-wise pattern of co-fluctuation. Overall, our work opens up the potential for future studies to identify the psychological and biological origins of this phenomenon of event synchronization in brains.
2 Results
Previous studies have identified brief, high-amplitude co-fluctuations in resting-state fMRI (R. Betzel et al., 2021; Esfahlani et al., 2020; Pope et al., 2021). The origins of these “events” are unclear. In this section, we explore the characteristics of these events in more detail using naturalistic stimuli (movie-watching) data from the Human Connectome Project (Van Essen et al., 2013) and a second, independently acquired dataset (Byrge & Kennedy, 2018). In the following sections, we describe results of these analyses. Specifically, we show that events synchronize across subjects at certain parts of the movie. We provide a tripartite classification scheme for the events that occur while subjects watch movies, as follows. Synchronous boundary events occur at the end of movie scenes (and beginning of rest blocks). Synchronous movie events occur during the movie, and asynchronous events do not synchronize across subjects. We show that boundary events exhibit reproducible and distinct spatiotemporal characteristics, as well as a unique activation pattern, and finally we observe a strong positive relationship between the similarity of time-locked co-fluctuation patterns and the propensity for those time-locked frames to involve synchronous events.
2.1 Events occur at the end of movie scenes
We first aimed to determine whether events synchronize during movie watching and, if so, when in the movies these synchronous events occurred. To investigate this, we estimated edge time series for every subject in every movie, calculated the co-fluctuation amplitude (root sum square of all edges at every time point; RSS), and used a previously described algorithm to detect event frames whose RSS exceeded that of a null distribution (R. Betzel et al., 2021). This resulted in an “event time series” (Fig. 1a). Two of the authors (J.T. and R.B.), without reference to event time series, then manually coded and hemodynamically convolved a time series to indicate the frames at which each movie scene ended and rest blocks began (Fig. 1b; see Methods for details on the structure of this naturalistic data, and our coding protocol). Authors were blinded to brain data during the coding procedure. We then “stacked” event time series and discovered an alignment between frames when many subjects had events and the indices of movie scene endings (Fig. 1c).
Events occurs at the end of movie scenes. (a) We compute edge time series (top), take the root sum square (RSS) to derive the co-fluctuation amplitude time series. Some of the peaks in co-fluctuation amplitude are determined to be events using a previously published statistical method (Betzel et al., 2021). The green line represents the binary event time series created from this method. (b) Schematic showing that movie scene endings were manually coded by two individuals authors (J.C.T. and R.F.B.), and then convolved with the hemodynamic response function (HRF). Example of HRF-convolved movie scene endings (orange) plotted below the relevant co-fluctuation amplitude time series. Note how many of the events from the event time series line up with movie scene endings. (c) Here, we “stack” the event time series across all subjects in a representative scan to illustrate that subjects tend to have events at the same moment in time, and that the moments when subjects synchronize their events often correspond to movie scene endings. (d) Here, we show this alignment statistically by cross correlating each subject’s event time series with the HRF-convolved movie scene endings, providing a Pearson correlation value for different time lags of the movie scene endings. We plot this separately for each of four movies. The thin lines represent the correlations across time lags for one subject and one movie scene ending. The bold lines represent the mean across movie scene endings and subjects. Positive time lags denote alignments where the events occurred after the movie scene endings. (e) Boxplots showing the distribution of cross correlation values for the time lag of 5 seconds. Each of these distributions are greater than expected by chance (one-sample t-test ). (f) Plot showing the time-lagged correlation between the indices of movie scene endings, and mean RSS across subjects (first plot) per scan (, all ). Plot showing the time-lagged correlation between the indices of movie scene endings, and the number of events per time point (second plot), with one line for each movie (, all ). (g) Plot showing the probability of an event occurring near a movie scene ending (movie scene endings centered on zero). Black line is the probability over all scans, and the gray lines are the probability per scan comparing the peak value in the probability of having an event at a time lag of 5 seconds with the probability at all other time lags considered (between −20 and +40 seconds) using a two-sample t-test (). (h) Plot showing the mean co-fluctuation amplitude (RSS) near movie scene endings. Black line is the mean across all scans, and the gray lines are the mean per scan.
Events occurs at the end of movie scenes. (a) We compute edge time series (top), take the root sum square (RSS) to derive the co-fluctuation amplitude time series. Some of the peaks in co-fluctuation amplitude are determined to be events using a previously published statistical method (Betzel et al., 2021). The green line represents the binary event time series created from this method. (b) Schematic showing that movie scene endings were manually coded by two individuals authors (J.C.T. and R.F.B.), and then convolved with the hemodynamic response function (HRF). Example of HRF-convolved movie scene endings (orange) plotted below the relevant co-fluctuation amplitude time series. Note how many of the events from the event time series line up with movie scene endings. (c) Here, we “stack” the event time series across all subjects in a representative scan to illustrate that subjects tend to have events at the same moment in time, and that the moments when subjects synchronize their events often correspond to movie scene endings. (d) Here, we show this alignment statistically by cross correlating each subject’s event time series with the HRF-convolved movie scene endings, providing a Pearson correlation value for different time lags of the movie scene endings. We plot this separately for each of four movies. The thin lines represent the correlations across time lags for one subject and one movie scene ending. The bold lines represent the mean across movie scene endings and subjects. Positive time lags denote alignments where the events occurred after the movie scene endings. (e) Boxplots showing the distribution of cross correlation values for the time lag of 5 seconds. Each of these distributions are greater than expected by chance (one-sample t-test ). (f) Plot showing the time-lagged correlation between the indices of movie scene endings, and mean RSS across subjects (first plot) per scan (, all ). Plot showing the time-lagged correlation between the indices of movie scene endings, and the number of events per time point (second plot), with one line for each movie (, all ). (g) Plot showing the probability of an event occurring near a movie scene ending (movie scene endings centered on zero). Black line is the probability over all scans, and the gray lines are the probability per scan comparing the peak value in the probability of having an event at a time lag of 5 seconds with the probability at all other time lags considered (between −20 and +40 seconds) using a two-sample t-test (). (h) Plot showing the mean co-fluctuation amplitude (RSS) near movie scene endings. Black line is the mean across all scans, and the gray lines are the mean per scan.
To confirm this observation statistically, we then performed several complementary tests. First, we correlated this “event time series” with the time series of hemodynamically convolved indices of movie scene endings (Fig. 1d). We found that event time series were correlated with the movie-scene endings (Fig. 1e; mean correlation per movie , , , respectively). When we averaged event time series across subjects to obtain a group-level and pseudo-continuous estimate of event time series, we found that the fraction of subjects exhibiting an event at any given instance was maximally correlated with movie-scene endings at a lag of 5 seconds (Fig. 1f; correlation at lag: , , , respectively, all p-values ). This effect is also evident in the raw and unthresholded co-fluctuation amplitude (Fig. 1f; correlation with mean RSS across subjects at lag: , , , respectively, all p-values ). Finally, we found that these effects resolved to an increased probability of having an event at a lag of 5 seconds from movie-scene endings (Fig. 1g; probability of having an event ; compared with probability at other nearby time points; two-sample t-test ).
We note that in a supplemental analysis we found this same effect with data where no global signal regression was performed. In addition, we showed that this same effect is observed across three different similarity metrics (Pearson correlation, Spearman correlation, and normalized mutual information; Fig. S4).
Importantly, this result was replicated in an independently collected, processed, and parcellated data set. Instead of movie scenes, this data set presented participants with movie trailers. J.T. and R.B. again manually coded and hemodynamically convolved movie trailer endings, and compared these with the stacked event time series (Fig. S1a). We then computed the number of events within a window of 10 seconds on either side of each movie trailer ending and compared this with a null model where this window was circularly shifted 100 times. We found that there were more events near movie trailer endings than should be expected by chance (Fig. S1b,c; p-values for each movie: , , , ). Note that the 10-second window was selected to include the peak lag of 5 seconds, but also was broad enough to account for inter-individual differences in timing.
Taken together, these results suggest that the endings of movie scenes (or trailers) correspond to time points when events are likely to occur in many subjects.
2.2 Events synchronize within movie scenes
In the previous section, we demonstrated that the ends of movie scenes coincide with high-amplitude co-fluctuations and are often categorized as events. It is unclear, however, if synchronized co-fluctuations also occur within individual movies. Here, we develop a statistical test to determine whether there are other points in time, specifically during the movies, when events occur coincidentally across subjects.
To address this question, we calculated the group-averaged event time series by summing across participant-level event time series. Each frame of this group time series indicated the number of subjects exhibiting an event at that instant. To identify temporally coincident events, we compared the observed group-averaged event time series with a null distribution estimated using resting-state scans rather than movie-watching (Fig. 2a). In the resting data, because subjects’ fMRI BOLD timecourses are not temporally locked to a movie stimulus, any observed synchronization is due to chance fluctuations. We also created a null model by circularly shifting event time series while subjects watched movies in order to maintain event number and relative timing while breaking alignment with the movie stimulus. However, we ultimately chose the rest null model after determining it was more conservative. That is, while both null models exhibited significantly fewer events than the intact (unshifted) movie data, we observed a greater number of inter-subject events using the resting-state null model than in the circularly shifted movie-watching data (Fig. S9; two-sample t-test ).
Boundary events exhibit distinct spatiotemporal characteristics. In the main text, we described methods for detecting events and partitioning frames into categories based on when they occurred during the movie and the number of subjects exhibiting temporally coincident events. To detect coincident (synchronous) events, we compared event counts at each movie frame with a null distribution estimated from resting-state data. (a) Percentage of subjects with events at each time point during resting-state scans. (b) Percentage of subjects with events at each time point during movie data, with frames labeled according to whether events were synchronous and occurred at movie boundaries (boundary event) or within a movie (movie event) or were not synchronized across individuals (asynchronous). Note that given the size of this cohort (), at least one participant exhibited an event at every frame. In principle, there could exist frames where no participant exhibited an event, necessitating a fourth category of time point (non-events). (c) Co-fluctuation amplitudes grouped by event type. (d & e) Global similarity of whole-brain co-fluctuation patterns grouped by event type (within each type, subject order is identical; mean across scans; see methods for details). (f) Mean co-fluctuation pattern for boundary events. (g) Typical temporal profile (co-fluctuation amplitude) for each event type (all events locked to their respective peaks). The asterisk refers to a statistically significant difference between the distributions.
Boundary events exhibit distinct spatiotemporal characteristics. In the main text, we described methods for detecting events and partitioning frames into categories based on when they occurred during the movie and the number of subjects exhibiting temporally coincident events. To detect coincident (synchronous) events, we compared event counts at each movie frame with a null distribution estimated from resting-state data. (a) Percentage of subjects with events at each time point during resting-state scans. (b) Percentage of subjects with events at each time point during movie data, with frames labeled according to whether events were synchronous and occurred at movie boundaries (boundary event) or within a movie (movie event) or were not synchronized across individuals (asynchronous). Note that given the size of this cohort (), at least one participant exhibited an event at every frame. In principle, there could exist frames where no participant exhibited an event, necessitating a fourth category of time point (non-events). (c) Co-fluctuation amplitudes grouped by event type. (d & e) Global similarity of whole-brain co-fluctuation patterns grouped by event type (within each type, subject order is identical; mean across scans; see methods for details). (f) Mean co-fluctuation pattern for boundary events. (g) Typical temporal profile (co-fluctuation amplitude) for each event type (all events locked to their respective peaks). The asterisk refers to a statistically significant difference between the distributions.
Comparing the observed event time series with the null distribution of subjects at rest, we found evidence of synchronous events occurring, as expected, at movie boundaries (we refer to these as “synchronous boundary events” or simply “boundary events”); but also within movie scenes (we refer to these as “synchronous movie events” or “movie events”; we controlled for multiple comparisons by fixing the false discovery rate to , ). We also defined a third category of time point—those for which the number of observed events across subjects was consistent with or less than that of the null distribution (referred to as “asynchronous events”; Fig. 2b).
2.3 Boundary events exhibit reproducible and distinct spatiotemporal characteristics
Given this tripartite classification scheme, we asked whether the three event types exhibited distinguishable characteristics. First, we tested whether boundary events exhibited dissimilar amplitudes than the other types. We found that, on average, boundary events had greater RSS than both movie and asynchronous events (Fig. 2c; two-sample t-test ).
Next, we asked whether the whole-brain co-fluctuation patterns expressed during events were dissimilar across event types. To do this, we calculated the similarity (bivariate linear correlation) between all pairs of detected events. Then, we averaged these scores by subject and event type. For every pair of subjects, this yielded three similarity scores—one per event type (Fig. 2d,e). For a given event type (indicated by the blocks shown in Fig. 2d), we compared within- and between-event type similarity. Intuitively, this corresponds to comparing elements in the diagonal blocks with the off-diagonal blocks. We found that “boundary events” were more similar to other boundary events than to other event types (Fig. 2e; two-sample t-test ), suggesting that boundary events represent a distinct category of events, dissimilar from the other two, which are largely indistinguishable from one another.
Given that boundary events were found to be similar to one another, we took the mean across boundary events to approximate the general edge-wise co-fluctuation pattern during boundary events (Fig. 2f). Notably, boundary events displayed stronger co-fluctuation within many systems, specifically interactions of the central visual system with salience and control networks (spin test, false discovery rate fixed at , ; Fig. S5g).
We then examined the local temporal structure around events. Briefly, this involved temporally aligning every instance of all event types to their respective peaks (Fig. 2g). We found, in agreement with the previous RSS analysis, that at their peaks, event types were stratified based on their amplitudes, with boundary events exhibiting greater RSS than the other event types. Interestingly, we also found off-peak effects. That is, the frames before and after peaks exhibited a similar relationship, suggesting that the frames immediately before and after boundary events also exhibited greater RSS than other event types (searched 10 frames on either side of event peak, two-sample t-test Bonferroni corrected for all 21 frames).
Critically, all of these effects were directly replicated in a second dataset (Fig. S2). In addition, we found that the mean co-fluctuation pattern for each event type also replicated (Fig. S3). We also found that time-averaged FC is mostly driven by synchronous movie events in the Human Connectome Project data, and that this effect is likely due to synchronous movie events containing more frames per scan than other event types (Fig. S8; two-sample t-test ) but this result did not replicate in the Indiana University data set (Fig. S8e-g).
We also found evidence that the different event types contained different amounts of individualized information. Specifically, we found that in aggregate asynchronous events contained the most individualized information (Fig. S6a paired-sample t-test ). On the other hand, when we considered individual frames, we found that boundary events carry the most individualized information (Fig. S7; two-sample t-test ; see Fig. S6 & S7 for more information).
Collectively, these results suggest that timing of events and their synchronicity across individuals shape their amplitude, co-fluctuation pattern, identifiability, and temporal evolution.
2.4 Activation patterns during boundary events
To this point, we have analyzed co-fluctuations, which are defined at the level of edges (pairs of brain regions). While the calculation of edge-time series from activations is straightforward, obtaining the activation pattern underpinning a given co-fluctuation matrix is not. This is because every co-fluctuation matrix could have been generated by two different patterns of activity that differ in their sign at every node. For example, the product of two positively valued nodal activations yields a positive co-fluctuation. However, had the activation amplitude been identical but negatively valued (a deactivation) we would still obtain an identical co-fluctuation. Here, we investigate the activation patterns that underlie co-fluctuations during events, focusing specifically on their configuration at movie boundaries.
First, we identified brain regions whose activity significantly increased or decreased during boundary events compared to all other time points using Pearson correlation (Fig. 3a,b; as many as 227 scans significant per region, Bonferroni corrected for number of scans tested ). Interestingly, we found that these correlations were highly system-specific, with increased activation among control b and salience/ventral attention b nodes and decreased activation in central visual (spin test; Bonferroni corrected ; Fig. 3b & Fig. S10). We also expanded our activation analysis to include subcortical and cerebellar regions of interest (Buckner et al., 2011; Choi et al., 2012; Fischl, 2012; Yeo et al., 2011). We found decreased activation in thalamic regions that are associated, based on their functional connectivity, with cortical control networks (Fig. 3c,e). We also found changes in activation patterns among cerebellar regions, specifically those associated with default mode, control, and dorsal attention networks (Fig. 3c,e; as many as 151 time series significant per region; Bonferroni corrected for number of scans tested ).
boundary events correspond to activation of control, and salience systems, and deactivation of visual systems. (a) Plot of mean system activity for a window of time. The black line represents an index of when a boundary event occurred. Notice the decrease in visual activation and an increase in the activation of salience and control systems. (b) Mean correlation values (across subjects) between regional activity and boundary event indices organized according to brain system (indices of brain system are color coded on the left of the plot). A space-preserving spin test was used to test if any of these brain systems tended to have a higher or lower concentration of correlation values than expected by chance. Control b, salience b, and central visual systems passed this test (). Systems that did not pass this test are shown shaded in gray. (c) Mean correlation values (across subjects) per subcortical and cerebellar region. Regions are organized mainly into parcellations for striatum, thalamus, and cerebellum. (d & e) Plot of the correlation values per subject and node (cortical, and subcortical respectively). (f) The pattern of correlation values across nodes is significantly similar across subjects, when compared with a time-shifted null model where we circularly shifted the boundary event indices before correlating them with the nodal time series ().
boundary events correspond to activation of control, and salience systems, and deactivation of visual systems. (a) Plot of mean system activity for a window of time. The black line represents an index of when a boundary event occurred. Notice the decrease in visual activation and an increase in the activation of salience and control systems. (b) Mean correlation values (across subjects) between regional activity and boundary event indices organized according to brain system (indices of brain system are color coded on the left of the plot). A space-preserving spin test was used to test if any of these brain systems tended to have a higher or lower concentration of correlation values than expected by chance. Control b, salience b, and central visual systems passed this test (). Systems that did not pass this test are shown shaded in gray. (c) Mean correlation values (across subjects) per subcortical and cerebellar region. Regions are organized mainly into parcellations for striatum, thalamus, and cerebellum. (d & e) Plot of the correlation values per subject and node (cortical, and subcortical respectively). (f) The pattern of correlation values across nodes is significantly similar across subjects, when compared with a time-shifted null model where we circularly shifted the boundary event indices before correlating them with the nodal time series ().
Importantly, we found that these correlation patterns were highly stable across individuals, both at the level of cortex as well as at the level of the subcortex and cerebellum (Fig. 3d,e; mean correlation , compared against correlation patterns found when using randomly permuted boundary event indices; ).
In summary, these results suggest that, although co-fluctuation patterns could, in principle, arise from a degenerate set of activity patterns, in the case of boundary events, they are underpinned by a single mode of activation and deactivation, involving a specific constellation of brain systems and regions.
2.5 Time-locked movie events exhibit distinct co-fluctuation patterns
To this point we have shown that movie boundaries tend to elicit high-amplitude co-fluctuations that are synchronous across individuals and exhibit distinct spatiotemporal characteristics that distinguish them from other types of events. On one hand, perhaps this is to be expected; movie boundaries correspond to periods during the movie with similar audiovisual features, for example, black or darkened screen and the absence of auditory stimuli. However, we have largely overlooked synchronous events that occur within the context of a single movie segment—the so-called “synchronous movie events.” In this section, we take advantage of the repeated presentation of the same movie scene across multiple scans to characterize the stable properties of synchronous and time-locked events that occur within the same movie.
To address this question, we first extracted frames corresponding to the same movie scene presented across four independent scans (a total of 516 viewings; Fig. 4a). Using the previously described algorithm, we detected synchronous events. Briefly, this involved creating a null distribution for the percentage of subjects/viewings we should expect to have events at the same time point by chance (using resting-state data, Fig. S9). This null distribution allows us to statistically test whether a given percentage of subjects/viewings with events are synchronizing more than we should expect by chance. Given that we were focusing on fMRI data where subjects watched the same movie scene, there were no boundaries between movie scenes. This meant that any events that were shown to be significantly synchronized across subjects/viewings occurred during the movie scene and should be considered movie events.
Time-locked movie events have similar co-fluctuation patterns. (a) Schematic showing that all four scans ended with the same movie clip, and how we temporally aligned the time series data from the four different presentations of this clip. (b) Histogram on the left shows the null distribution of subjects at rest. The plot on the right shows the percentage of viewings with events in the actual movie-watching data across all viewings of the same movie (in blue). The dashed line indicates when this number is significantly greater than chance. We highlighted and numbered three peaks that we will focus on in further analyses. (c) This plot shows the 129 subjects who watched this movie scene four times. The values in this matrix show how many times a given subject had an event during each frame, where four is the maximum number possible. Three areas stand out in this matrix, and as the plot below the matrix shows, these areas correspond to frames where as many as 15 subjects had an event every time they watched this movie scene. (d, e & f) These three matrices show the mean co-fluctuation patterns (across subjects) at the frames highlighted in other plots. (g) This matrix shows the main differences between the first peak and the second peak in their system by system interactions. (h) This matrix shows the main differences between the second peak and the third peak in their system by system interactions. (i) Plot showing the positive linear relationship between percentage of viewings with events per time point, and the mean similarity of co-fluctuation patterns (across subjects). (j) Plot showing the positive linear relationship between the mean co-fluctuation amplitude (across subjects), and the mean similarity of co-fluctuation patterns (across subjects). (k) Mean similarity of co-fluctuation patterns (across subjects) plotted across time. Three frames/peaks are highlighted. (l) Matrix showing the similarity of co-fluctuation patterns for the three time-locked frames. (m) Boxplots showing the data in the previous figure divided into similarity within and between the highlighted peaks (first, second, & third). Each peak is significantly more similar to itself than to the other peaks. (n) Screen shots from the movie scenes color coded to correspond with the time series displayed in b, c, & k. See Supplementary Movie for a video of brain networks, events, and this movie side-by-side. The asterisk refers to a statistically significant difference between the distributions.
Time-locked movie events have similar co-fluctuation patterns. (a) Schematic showing that all four scans ended with the same movie clip, and how we temporally aligned the time series data from the four different presentations of this clip. (b) Histogram on the left shows the null distribution of subjects at rest. The plot on the right shows the percentage of viewings with events in the actual movie-watching data across all viewings of the same movie (in blue). The dashed line indicates when this number is significantly greater than chance. We highlighted and numbered three peaks that we will focus on in further analyses. (c) This plot shows the 129 subjects who watched this movie scene four times. The values in this matrix show how many times a given subject had an event during each frame, where four is the maximum number possible. Three areas stand out in this matrix, and as the plot below the matrix shows, these areas correspond to frames where as many as 15 subjects had an event every time they watched this movie scene. (d, e & f) These three matrices show the mean co-fluctuation patterns (across subjects) at the frames highlighted in other plots. (g) This matrix shows the main differences between the first peak and the second peak in their system by system interactions. (h) This matrix shows the main differences between the second peak and the third peak in their system by system interactions. (i) Plot showing the positive linear relationship between percentage of viewings with events per time point, and the mean similarity of co-fluctuation patterns (across subjects). (j) Plot showing the positive linear relationship between the mean co-fluctuation amplitude (across subjects), and the mean similarity of co-fluctuation patterns (across subjects). (k) Mean similarity of co-fluctuation patterns (across subjects) plotted across time. Three frames/peaks are highlighted. (l) Matrix showing the similarity of co-fluctuation patterns for the three time-locked frames. (m) Boxplots showing the data in the previous figure divided into similarity within and between the highlighted peaks (first, second, & third). Each peak is significantly more similar to itself than to the other peaks. (n) Screen shots from the movie scenes color coded to correspond with the time series displayed in b, c, & k. See Supplementary Movie for a video of brain networks, events, and this movie side-by-side. The asterisk refers to a statistically significant difference between the distributions.
In the event time series, three peaks stood out. These peaks corresponded to points in time when many subjects exhibited an event (Fig. 4b), and many subjects’ co-fluctuation patterns were highly similar (Fig. 4k) and some individuals who watched the movie scene four times exhibited events upon each viewing (Fig. 4c). These three peaks corresponded to specific parts of the movie scene involving a couple getting married, a subsequent viewing of a young girl (presumably the daughter of the previously married couple), and an email message (accompanied by a ringing sound indicating a message alert; Fig. 4n; for more on the content of the movies, see Fig. S12). We also created a video that shows this movie scene alongside both the mean co-fluctuation pattern (across subjects/viewings) and a plot of these peaks (see Supplementary Movie).
Next, we examined the co-fluctuation patterns during these three peaks (Fig. 4d,e,f). Interestingly, we found that these co-fluctuation patterns were dissimilar to one another, driven by system-specific differences involving the dorsal attention and temporo-parietal networks (Fig. 4g,h). We further analyzed these co-fluctuation patterns to see if they were meaningfully distinct across subjects. We performed the same analysis that we used in a previous section (to identify the similarity of boundary events to one another) in order to identify if different instantiations of the same peak (from different subjects, or viewings) were highly similar (Fig. 4l). We found that the co-fluctuation patterns of these three peaks could be reliably distinguished from one another based on which peak they corresponded to (Fig. 4l,m; ).
Upon reflection, this pattern—wherein peaks were dissociable from one another based on co-fluctuation patterns and event status—represents the tail of a more general phenomenon in which inter-subject similarity is positively associated with both the percentage of subjects/viewings with an event and the mean across subject co-fluctuation amplitude (, ; , , respectively; Fig. 4i,j). In other words, when subjects align in terms of the global feature of co-fluctuation amplitude (or events), they are also more likely to align in more local features found in the specific co-fluctuation pattern. For this reason, the local features of the co-fluctuation patterns at these three peaks are more likely to be similar across subjects given that these peaks involved synchronization at the global scale of events.
Finally, we also identified an additional peak in the event time series where many subjects synchronized. This peak occurred at a part of the movie where a couple was embracing for a kiss. Additionally, during this peak, co-fluctuation patterns were similar (Fig. 4k) across viewings. Interestingly however, subjects were not as likely to have an event when watching this movie scene multiple times (Fig. 4c).
We note that although previous studies have shown that viewing movies multiple times can result in an anticipation effect wherein regional neural signals temporally shift to earlier time points (e.g. C. S. Lee et al., 2021), we did not find this same effect with global measures like RSS amplitude or events (Fig. S11).
In summary, these results posit a link between the similarity of co-fluctuation patterns across subjects and the propensity for those patterns to occur synchronously across individuals. Further, it reveals repeatable, fine-scale, and spatiotemporally dissociable event structure within movie scenes.
3 Discussion
Previous studies have shown that spontaneous resting-state fMRI data exhibited “events”—brief, high-amplitude, and network-level co-fluctuations. The drivers of events remain unclear. Here, we investigate this question using data acquired while subjects passively viewed movie scenes (or trailers) in the scanner. We find evidence for three types of events: boundary events, movie events, and asynchronous events. We show that boundary events are likely to occur across subjects during the boundaries between sequential movie scenes (or trailers). Further, we show that these boundary events are of greater amplitude than non-boundary events, exhibit similar co-fluctuation patterns across subjects, and follow distinct temporal trajectories. Interestingly, boundary events also carry the most subject-specific information of any event type when considered individually although asynchronous events carry more subject-specific information when sub-sampled to generate FC per event types (by taking mean of a sample of events for each type). At the nodal level, boundary events are underpinned by a distinct pattern of activity, corresponding to deactivation of visual areas and the activation of control and ventral attention networks. In addition to showing that boundary events are distinct from other event types in many of their features, we show that movie events (synchronous events that occur during the movie) share features that are time-locked to the movies that subjects are watching. That is, when a movie scene elicits an event in many subjects, the whole-brain co-fluctuation patterns of these subjects tend to be similar. Additionally, we found that this specific pattern is a feature of a more general phenomenon wherein inter-subject synchrony in events is associated with higher inter-subject similarity in co-fluctuation patterns.
3.1 Movie boundaries reliably elicit high-amplitude co-fluctuations
Events explain a large fraction of variance in static FC and can improve subject identification and brain-behavior correlations (Cutts et al., 2022; Esfahlani et al., 2020). They are individualized (R. Betzel et al., 2021), are correlated with quotidian variability in hormones across the menstrual cycle (Greenwell et al., 2021), can be used to distinguish healthy controls from individuals with autism (Esfahlani et al., 2021), and can arise spontaneously in networks whose underlying anatomical structure is modular (Pope et al., 2021). However, the timing of events—why they occur when they do—is not understood. Resting-state datasets—the focus of most previous edge time series papers—are poorly suited for addressing this question; the unconstrained nature of rest and the absence of any temporal correlations across participants makes it difficult to establish possible event drivers.
Here, we leverage naturalistic movie-watching data (Sonkusare et al., 2019), (Finn, 2021; Maguire, 2012) to begin addressing this question. We find evidence that, while events occur at all times during movies, they reliably and synchronously occur at the end of movie scenes. This observation is consistent with several recent studies, which link activations and inter-subject synchronization at the boundaries of movies to memory (Hahamy et al., 2022; H. Lee & Chen, 2021; Masís-Obando et al., 2022), attention processes (Levakov et al., 2022), general cognition (Rajimehr et al., 2022), and narrative comprehension generally (Baldassano et al., 2018; Geerligs et al., 2022; Masís-Obando et al., 2022; Schwan et al., 2000; Speer et al., 2007).
Much of the relevant research here is related to the event segmentation theory (Kurby & Zacks, 2008). It is especially important to clarify the similarities and dissimilarities between our work and this research because the term “events” means different things for edge-centric analyses like ours (R. F. Betzel et al., 2022; R. Betzel et al., 2022; Esfahlani et al., 2020) than it does for the event segmentation theory. In edge-centric work like our own, the term “events” refers to a set of moments when many regions of the brain co-fluctuate. In contrast, in the event segmentation theory, the term “events” refers to perceptually defined chunks of experience. This research often focuses on the moments when one perceptually defined chunk of experience transitions to another perceptually defined chunk of experience.
Why these edge-centric events reliably occur at the boundaries of movie scenes remains unclear. One possibility is that boundary events serve as brain-based markers for the discretization of experience (Landau et al., 2018; Miller, 1956). That is, that boundary events signify the ending of one subjectively and perceptually defined segment and the beginning of another (Kurby & Zacks, 2008; Sargent et al., 2013; Shin & DuBrow, 2021). In this way, and purely by coincidence, edge-centric “events” and the “events” of the event segmentation theory might be related. If so, this adds to the literature on the event segmentation theory by showing that these boundaries between perceptually defined events might also share features of edge-centric events, for example, accounting for more of the variance in static functional connectivity.
Indeed, the replication of our main results might support this connection. In the main dataset, movie boundaries occur with a 20 second block of rest, but in the replication dataset the boundaries do not contain a resting block. Interestingly, we find that the events associated with the boundaries with the resting block occur at a 5 second offset from the end of the movie scene, whereas the events associated with the instantaneous boundaries (with no rest block) do not display this offset. This might suggest that when there is not an immediate change in the subject of the perceptually defined segment, these events are slightly delayed because the perceptual boundary can be defined more loosely. Although appealing, directly testing this hypothesis necessitates the collection of additional data to confirm that movie boundaries align with participants’ subjectively defined segment boundaries (Jafarpour et al., 2022).
Another possibility is that synchronous events arise from perturbations to a dynamical system. Many studies have shown that the temporal evolution of brain activity can be modeled as a dynamical system. If these systems are broadly similar across individuals, we might expect that driving them with the same time-varying inputs (audiovisual stimuli) will yield similar response profiles. This partially explains inter-subject correlations during naturalistic paradigms. We can speculate that the ends of movies, which are often typified by lower levels of luminance and sound volume, represent particularly evocative or arousing stimuli so that the shared response is brain-wide and initiates an event. Indeed, previous studies have shown that state of arousal is a powerful modulator of brain activity (Chang et al., 2016; Gonzalez-Castillo et al., 2022) and connectivity (Allen et al., 2014; W. J. Lee et al., 2022), suggesting that it may also play a role in event induction and should be investigated further in future studies.
We note that these hypotheses—experience segmentation and perturbations to dynamical systems—represent only two possible explanations for boundary events and are, themselves, not mutually exclusive. Additionally, we note that while movie scene boundaries are factors that explain some synchronous high-amplitude co-fluctuations, we also observe synchronous events at other points within movies. The underlying mechanisms that support “movie events” may be similar to boundary events, although stimuli during the movie are much richer and complex (Hasson et al., 2010). Future studies should focus on linking this second category of events with other features, for example, movie annotations or eye-tracking data as a proxy for arousal.
Finally, the co-fluctuation patterns during these synchronous boundary events were found to exhibit similarity to one another both within the main dataset (Fig. 2d) as well as between the main dataset and the replication dataset (Fig. S2e,f & Fig. S3). The main features of these co-fluctuation patterns involved increased edge strength between the central visual system and control systems (a & b). We found that underlying this co-fluctuation pattern was a distinct pattern of activation involving the activation of control systems and the de-activation of visual systems (Fig. S10a,b) as well as the de-activation of thalamic regions alongside the activation and de-activation of different cerebellar regions (Fig. 3c).
The de-activation of visual systems at the end of movie scenes is easy to explain by changes in the luminance accompanying many of the movie scene/trailer endings where the screen would often darken. However, the consistent activation of control systems and cerebellar systems, alongside de-activation of thalamic and other cerebellar regions, is less easy to explain by low-level sensory features of the movie stimuli. Perhaps these regions are involved in the discretization of experience into distinct segments theorized by the event segmentation theory (Kurby & Zacks, 2008). Indeed, past studies have shown increased activation in the posterior cingulate and precuneus at perceived event boundaries (Speer et al., 2007), and both of these regions are members of the control system that were activated in response to these boundaries in our data. Although subcortical systems like the locus coeruleus have been implicated in the reset of these perceptual segments (Zacks et al., 2007), the implication of thalamic and cerebellar contributions to this process is less well explored. Future studies should explore the role of thalamic and cerebellar contributions to this process more directly.
3.2 Inter-subject synchronization and events
Numerous previous studies have utilized inter-subject synchronization in brain activity to examine the roles of various brain regions in comprehending movies (Chen et al., 2008; Hasson et al., 2010, 2004; Simony et al., 2016). In this research, we employ an innovative edge-centric method (Esfahlani et al., 2020) that maintains the native temporal resolution of any time series to detect time-varying brain networks during movie-watching. This technique is known to emphasize “events,” or high-amplitude co-fluctuations, which manifest as spikes in brain-wide edge amplitude. Consequently, these instances may signify substantial communication events throughout the brain.
Although these events are determined by the statistical properties of an individual subject’s brain, we discovered that all subjects’ brain networks exhibit the greatest similarity during events while viewing movies. In consequence, subjects not only synchronize in terms of their coarsely defined whole-brain edge amplitude, but they also synchronize in the fluctuating strength of inter-regional connectivity. Importantly, this synchronized pattern of inter-regional connectivity (co-fluctuation pattern) is time-locked to the movie-stimulus, suggesting that these patterns reflect brain processing of the ongoing movie. Taken together, these results suggest that the phenomenon of brain synchronization at the whole-brain level during naturalistic viewing is generally connected to the occurrence of events.
3.3 Event timing is not explained by sampling variability
Several studies have suggested that events arise not from any neurocognitively meaningful mechanism, but simply from sampling variability around a fixed correlation structure (Ladwig et al., 2022; Novelli & Razi, 2021). That is, given a static FC matrix, nodal and edge time series must have a specific set of properties, including events. This is an important concern, and one that echoes other ongoing and unresolved debates in the network and cognitive neuroscience literature (Laumann et al., 2017; Liegeois et al., 2017; Lurie et al., 2020).
Our results speak directly to this controversy. Specifically, if events arise only due to stochastic fluctuations, then their occurrences should be independent and uncorrelated across individuals. However, we find that this is not the case and that, in line with previous studies (Esfahlani et al., 2020), the timing of events within a scan session is correlated across individuals. These events cannot obviously be attributed to “rest blocks” that are interspersed between movie scenes, as we also find evidence of correlated events within movies.
These observations contribute to and enrich the ongoing dialogue surrounding exactly what features of brain networks are stationary versus non-stationary. Future studies—both empirical and in silico—should continue to investigate these and related questions.
3.4 Future directions
There are several ways that the results of this study could be extended in the future. For instance, we examined naturalistic stimuli (passive movie-watching). We found robust evidence of inter-subject correlations, specifically the timing of events, both during and at the end of movie scenes. While it is clear that subjects synchronize events, the precise causes remain unclear. One way to gain insight into these causes involves leveraging the movie annotations that accompany the imaging data. That is, to link the timing of events, synchronized or otherwise, to the timing of particularly salient features (e.g. objects and actions) occurring in the movie (Hanke et al., 2014). Additionally, future studies with more specified and explicit task design could also be performed to adjudicate between competing hypotheses about the origins of events. For example, online experience sampling during acquisition could be used to better understand the subjective experiences that co-occur with events (Kucyi et al., 2021).
Another way to extend these results would be to examine multi-modal recordings made at, potentially, faster timescales, for example, scalp or intracranial EEG or MEG (R. F. Betzel et al., 2019; Chu et al., 2012; Scheid et al., 2021). Our work here, and related studies, has focused almost exclusively on fMRI data. However, fMRI has a number of known limitations, most notably it exhibits relatively poor temporal resolution, which may obscure rapidly occurring neural processes, for example, fine-grained temporal structure that occurs just prior to or following an event (Abbas et al., 2019; Fisch et al., 2009; Krohn et al., 2021). Future studies should investigate inter-synchrony of events using imaging modalities that are acquired at faster rates.
Here, we focus on empirical recordings. However, recent studies have shown that events can be observed in synthetic data generated by dynamic models (Pope et al., 2021). This opens up several avenues for future work, including artificially “stimulating” in silico brains to induce events and, if we have two independent simulations, assessing whether simultaneous stimulation leads them to synchronize (Ritter et al., 2013). The simulation-based analyses also make it possible to investigate and test related hypotheses using data from non-human subjects, for example, macaque (O’Reilly et al., 2013).
One of our key findings was that events are temporally coincident during movie-watching and especially near the boundaries between movie trailers. However, participants consistently exhibit events throughout the scan, even if the timing is not coincident with other individuals. What are the origins of these “asynchronous events?” One possibility is that they reflect an intrinsic mode of rest that is decoupled from the ongoing task/movie stimuli (Cole et al., 2014). Future studies, likely with dedicated experimentation, are needed to clarify the distinction between asynchronous and synchronous events.
Another noteworthy point concerns how events were defined. Here, a time point was considered an event if global—that is, whole-brain—co-fluctuations exceeded a statistical criterion. We found that these global events reliably occurred during movie boundaries, which correspond to large shifts in movie content. Future studies should investigate locally defined events—for example, at the level of brain systems—which may be linked to more subtle variation in movie features.
Finally, while we find evidence of synchronous events, we note that synchronization does not imply that all subjects experienced an event at the same moment (just that a sufficiently large number did). An important open question is whether those subjects that did not experience an event are distinguished from their peers along other dimensions as well, for example, cognitive, clinical, demographic profiles. For example, perhaps subjects that experienced an event have better recall of related movie scenes than subjects that did not.
3.5 Limitations
There are several limitations of the current study. One notable issue is the use of fMRI data to infer brain activity. Functional imaging data are fundamentally an indirect (and slow) measure of the hemodynamic response to population-level activity. Follow-up studies should aim to reproduce these results using more direct assessments of activity, for example, intracranial EEG.
Additionally, there are several limitations associated with the movie data themselves. The first concerns the subjectivity with which individuals experience the movie stimuli. While the stimuli are identical across individuals, their experiences may not be, for example, participants may attend to different movie features at the same time point or draw on their personal history, leading to dissimilar experience. Without subjective post-scan reports or incorporation of additional data modalities, for example, eye-tracking, it remains challenging to uncover the drivers of synchronized events.
Furthermore, it is important to clarify that while previous studies have primarily focused on events during the resting-state, we focus here on events detected during passive movie-watching. While the statistical approach used to detect periods of high-amplitude co-fluctuations is identical, it remains unclear whether resting and movie events2 share common neurocognitive underpinnings. Future studies should investigate their differences and commonalities in more detail.
For instance, while past studies have shown that events during the resting-state can be divided into distinct clusters based on the spatial similarity of their co-fluctuation patterns (R. F. Betzel et al., 2022; R. Betzel et al., 2022; Greenwell et al., 2021; Ragone et al., 2023), here we cluster events into types based on their timing during movie-watching and show that this partitioning divides the events into types that show a moderate within-type preference in the spatial similarity of their co-fluctuation patterns. However, if we were to cluster events during movie-watching based on spatial similarity and not when they occurred in the movie, it might be the case that we could uncover greater diversity in terms of their spatial similarity. Future work should explore this possibility directly.
We also note that although we found a strong positive correlation between global synchronization in events (or RSS amplitude) and local synchronization in co-fluctuation patterns, there were notably exceptions to this pattern. For example, although the percentage of viewings where a subject had an event during the kiss (Fig. 4b, 35%) is less than when the email message comes (Fig. 4b, 50%), the inter-viewing similarity of co-fluctuation patterns is higher for the kiss (Fig. 4k, mean r = 0.15) than for the email message (Fig. 4k, mean r = 0.1). However, it is unclear whether or not this variability in the linear relationship between the global synchronization (events) and local synchronization (co-fluctuation patterns) is meaningful or is primarily driven by noise.
Lastly, there remain challenges related to data processing and analysis. Specifically, the procedures for disambiguating activations (first-order effects) from co-fluctuations (second-order effects) during movie-watching are not well defined. In blocked or event-related task design, regressors can be constructed to orthogonalize time series with respect to activations (Cole et al., 2019), making it possible to effectively remove activations as confounds prior to estimating connectivity or co-fluctuations. However, with movie-watching data, stimuli are presented in a naturalistic way, meaning that many stimuli overlap and co-occur, for example, the presence of a face is often correlated with speech, making it difficult to control for all possible stimuli. Moreover, the stimuli are presented in a continuous stream, similarly making it difficult to construct appropriate regressors. This remains an active and contentious area of research. Future methodological studies will address this potential issue. One particular concern unique to our findings is the presence of “rest blocks” in the HCP dataset. An alternative interpretation of our results is that because the transition from a “movie state” to “rest state” is accompanied by a short pause, it is the pause itself that induces a non-stationarity in the activations, yielding a boundary event.
4 Conclusion
In conclusion, this study represents an analysis of the synchronization of high-amplitude co-fluctuations, or “events,” while participants are watching the same movie in two separately collected, processed, and parcellated data sets. Our analysis suggests that events during movie-watching can be divided into three categories based upon their synchronization. Synchronous boundary events occur at the end of movie scenes, or trailers, and are distinct from the other event types in overall co-fluctuation pattern, amplitude, and temporal structure, as well as carrying the most subject-specific information at the level of individual frames. Synchronous movie events occur during the movie and their time-locked co-fluctuation patterns are highly similar across subjects. In fact, we found a general relationship between the degree of event synchronization across subjects and the across-subject similarity in co-fluctuation pattern. These two types of synchronous events suggest that events can be elicited by environmental stimuli in a similar manner across subjects. Asynchronous events also occur throughout the movie, suggesting an individualized element to the inducement of events by movies. Indeed, when considered together, asynchronous events also carry the most individualized information about subjects.
5 Materials and Methods
5.1 Human Connectome Project data
The Human Connectome Project (HCP) 7 T dataset (Van Essen et al., 2013) consists of structural magnetic resonance imaging (T1w), resting state functional magnetic resonance imaging (rsfMRI) data, and movie watching functional magnetic resonance imaging (mwfMRI) from 184 adult subjects. These subjects are a subset of a larger cohort of approximately 1200 subjects additionally scanned at 3 T. Subjects’ 7 T fMRI data were collected during four scan sessions over the course of two or three days at the Center for Magnetic Resonance Research at the University of Minnesota. Subjects’ 3 T T1w data were collected at Washington University in St. Louis. The study was approved by the Washington University Institutional Review Board, and informed consent was obtained from all subjects.
5.2 Demographics
We analyzed MRI data collected from subjects (77 female, 52 male), after excluding subjects with poor-quality data. Upon defining each spike as relative framewise displacement of at least 0.25 mm, we excluded subjects who fulfill at least 1 of the following criteria: greater than of time points spike, average framewise displacement greater than 0.2 mm; contains any spikes larger than 5 mm. Following this filter, subjects who contained all four scans were retained. At the time of their first scan, the average subject age was years, with a range from . 70 of these subjects were monozygotic twins, 57 where non-monozygotic twins, and 2 were not twins.
5.3 MRI acquisition and processing
A comprehensive description of the imaging parameters and image preprocessing can be found in Glasser et al. (2013) and in HCP’s online documentation (https://www.humanconnectome.org/study/hcp-young-adult/document/1200-subjects-data-release). T1w were collected on a 3 T Siemens Connectome Skyra scanner with a 32-channel head coil. Subjects underwent two T1-weighted structural scans, which were averaged for each subject (TR = 2400 ms, TE = 2.14 ms, flip angle = 8°, 0.7 mm isotropic voxel resolution). fMRI were collected on a 7 T Siemens Magnetom scanner with a 32-channel head coil. All 7 T fMRI data was acquired with a gradient-echo planar imaging sequence (TR = 1000 ms, TE = 22.2 ms, flip angle = 45, 1.6 mm isotropic voxel resolution, multi-band factor = 5, image acceleration factor = 2, partial Fourier sample = 7/8, echo spacing = 0.64 ms, bandwidth = 1924 Hz/Px). Four resting-state data runs were collected, each lasting 15 minutes (frames = 900), with eyes open and instructions to fixate on a cross. Four movie-watching data runs were collected, each lasting approximately 15 minutes (frames = 921, 918, 915, 901), with subjects passively viewing visual and audio presentations of movie scenes. Movies consisted of both freely available independent films covered by Creative Commons licensing and Hollywood movies prepared for analysis (Cutting et al., 2012). For both resting-state and movie-watching data, two runs were acquired with posterior-to-anterior phase encoding direction and two runs were acquired with anterior-to-posterior phase encoding direction.
Structural and functional images were minimally preprocessed according to the description provided in Glasser et al. (2013). 7 T fMRI images were downloaded after correction and reprocessing announced by the HCP consortium in April, 2018. Briefly, T1w images were aligned to MNI space before undergoing FreeSurfer’s (version 5.3) cortical reconstruction workflow. fMRI images were corrected for gradient distortion, susceptibility distortion, and motion, and then aligned to the corresponding T1w with one spline interpolation step. This volume was further corrected for intensity bias and normalized to a mean of 10000. This volume was then projected to the 2 mm 32k_fs_LR mesh, excluding outliers, and aligned to a common space using a multi-modal surface registration (Robinson et al., 2014). The resultant cifti file for each HCP subject used in this study followed the file naming pattern:
*_Atlas_MSMAll_hp2000_clean.dtseries.nii. Resting-state and moving-watching fMRI images were nuisance regressed in the same manner. Each minimally preprocessed fMRI was linearly detrended, band-pass filtered (0.008–0.25 Hz), confound regressed, and standardized using Nilearn’s signal.clean function, which removes confounds orthogonally to the temporal filters. The confound regression strategy included six motion estimates, mean signal from a white matter, cerebrospinal fluid, and whole-brain mask, derivatives of these previous nine regressors, and squares of these 18 terms. Spike regressors were not applied. Following these preprocessing operations, the mean signal was taken at each time frame for each node, as defined by the Schaefer 400 parcellation (Schaefer et al., 2018) in 32k_fs_LR space.
5.4 Indiana University data
5.4.1 Demographics
We analyzed MRI data collected from subjects (5 female, 24 male; 25 were right-handed). This cohort was male-dominant, as subjects were intended to serve as controls for a study in autism spectrum disorder, which is more common in men than women. At the time of their first scan, the average subject age was years.
5.4.2 MRI acquisition and processing
MRI images were acquired using a 3 T whole-body MRI system (Magnetom Tim Trio, Siemens Medical Solutions, Natick, MA) with a 32-channel head receive array. Both raw and prescan-normalized images were acquired; raw images were used at all preprocessing stages and in all analyses unless specifically noted. During functional scans, T2*-weighted multiband echo planar imaging (EPI) data were acquired using the following parameters: TR/TE = 813/28 ms; 1200 vol; flip angle = 60°; 3.4 mm isotropic voxels; 42 slices acquired with interleaved order covering the whole brain; multi-band acceleration factor of 3. Preceding the first functional scan, gradient-echo EPI images were acquired in opposite phase-encoding directions (10 images each with P-A and A-P phase encoding) with identical geometry to the EPI data (TR/TE = 1175/39.2 ms, flip angle = 60°) to be used to generate a fieldmap to correct EPI distortions, similar to the approach used by the Human Connectome Project (Smith et al., 2013). High-resolution T1-weighted images of the whole brain (MPRAGE, 0.7 mm isotropic voxel size; TR/TE/TI = 2499/2.3/1000 ms) were acquired as anatomical references.
All functional data were processed according to an in-house pipeline using FEAT (v6.00) and MELODIC (v3.14) within FSL (v. 5.0.9; FMRIB’s Software Library, www.fmrib.ox.ac.uk/fsl), Advanced Normalization Tools (ANTs; v2.1.0) (Avants et al., 2011), and Matlab_R2014b. This pipeline was identical to the GLM + MGTR procedure described in Byrge and Kennedy (2018).
In more detail, individual anatomical images were bias-corrected and skull-stripped using ANTs, and segmented into gray matter, white matter, and CSF partial volume estimates using FSL FAST. A midspace template was constructed using ANTs’ buildtemplateparallel and subsequently skull-stripped. Composite (affine and diffeomorphic) transforms warping each individual anatomical image to this midspace template, and warping the midspace template to the Montreal Neurological Institute MNI152 1 mm reference template, were obtained using ANTs.
For each functional run, the first five volumes (4 seconds) were discarded to minimize magnetization equilibration effects. Framewise displacement traces for this raw (trimmed) data were computed using fsl_motion_outliers. Following (Burgess et al., 2016; Byrge & Kennedy, 2018), we performed FIX followed by mean cortical signal regression. This procedure included rigid-body motion correction, fieldmap-based geometric distortion correction, and non-brain removal (but not slice-timing correction due to fast TR (Smith et al., 2013)). Preprocessing included weak high-pass temporal filtering (2000 seconds FWHM) to remove slow drifts (Smith et al., 2013) and no spatial smoothing. Off-resonance geometric distortions in EPI data were corrected using a fieldmap derived from two gradient-echo EPI images collected in opposite phase-encoding directions (posterior-anterior and anterior-posterior) using FSL topup.
We then used FSL-FIX (Salimi-Khorshidi et al., 2014) to regress out independent components classified as noise using a classifier trained on independent but similar data and validated on hand-classified functional runs. The residuals were regarded as “cleaned” data. Finally, we regressed out the mean cortical signal (mean BOLD signal across gray matter partial volume estimate obtained from FSL FAST). All analyses were carried out on these data, which were registered to subjects’ skull-stripped T1-weighted anatomical imaging using Boundary-Based Registration (BBR) with epi_reg within FSL. Subjects’ functional images were then transformed to the MNI152 reference in a single step, using ANTS to apply a concatenation of the affine transformation matrix with the composite (affine + diffeomorphic) transforms between a subject’s anatomical image, the midspace template, and the MNI152 reference. Prior to network analysis, we extracted mean regional time series from regions of interest defined as sub-divisions of the 17-system parcellation reported in Yeo et al. (2011) and used previously (R. F. Betzel et al., 2014, 2015; Byrge et al., 2015). Wakefulness during movie and rest scans was monitored in real time using an eye tracking camera (Eyelink 1000).
5.4.3 Naturalistic stimuli and coding boundaries
5.4.3.1 Human Connectome Project data
Movies consisted of concatenated movie scenes with 20 second blocks of rest between them. The movies scenes were sourced from both freely available independent films covered by Creative Commons licensing and Hollywood movies prepared for analysis (Cutting et al., 2012).
Both J.C.T. and R.F.B. manually coded the boundaries between the end of movie scenes and the beginning of rest blocks and confirmed consistency across codings. Additionally, J.C.T. confirmed that these codings aligned with the end of movie scenes and the beginning of rest blocks using the RGB values from a digitized version of the movies. The movie screen during rest blocks, on average, contains more black pixels (See Fig. 1b, e.g.). By indexing the number of black pixels and overlaying the coded boundaries, J.C.T. found an alignment between the beginning of rest blocks (where the number of black pixels increased) and coded boundaries.
5.4.3.2 Indiana University data
All movies were obtained from Vimeo (https://vimeo.com). They were selected based on multiple criteria. First, to ensure that movie trailers represented novel stimuli, we excluded any movie that had a wide theatrical release. Secondly, we excluded movies with potentially objectionable content including nudity, swearing, drug use, etc. Lastly, we excluded movies with intentionally startling events that could lead to excessive in-scanner movement.
Each trailer lasted between 45 and 285 seconds (approximately 1 to 5 minutes). Each movie scan comprised between four and six trailers with genres that included documentaries, dramas, comedies, sports, mystery, and adventure (see Table S1 for more details). Both J.C.T. and R.F.B. manually coded the boundaries between trailers and confirmed consistency across codings.
5.4.4 Functional connectivity
Functional connectivity (FC) measures the statistical dependence between the activity of distinct neural elements. In the modeling of macroscale brain networks with fMRI data, this usually means computing the Pearson correlation of brain regions’ activity time series. To calculate FC for regions and , then, we first standardize their time series and represent them as z-scores. We denote the z-scored time series of region as , where is the number of samples. The Pearson correlation is then calculated as:
In other words, the correlation is equal to the temporal average of two regions’ co-fluctuation.
5.4.5 Edge time series
We analyzed edge time series data. Edge time series can be viewed as a temporal decomposition of a correlation (functional connection) into its framewise contributions. Note that Pearson correlation is calculated as , where is the number of samples and is the z-scored transformation of the time series . If we omit the summation in our calculation of , we obtain a time series , whose elements index the instantaneous co-fluctuation between variates and . Here, we estimated the edge time series for all pairs of brain regions .
We analyzed edge time series using two distinct approaches.
5.4.5.1 Edge time series amplitude (RSS)
First, we calculated the amplitude at each frame as the root sum of squared co-fluctuations. That is, the amplitude at time was given by:
5.4.5.2 Whole-brain co-fluctuation patterns
The section way in which we analyzed edge time series was by considering the whole-brain co-fluctuation pattern at a given time point. That is, we focused on the co-fluctuation matrices, . The element in this matrix denoted the co-fluctuation magnitude between regions and at time , that is, . Although co-fluctuation matrices are not correlation matrices, they can nonetheless be analyzed using similar network science tools.
5.4.6 Global similarity matrix
We assessed the similarity of events between subjects and types (boundary, movie, and asynchronous). The procedure for doing so involved two steps. First, for a given pair of subjects, and , with and events, respectively, we calculated the similarity matrix between all pairs of events. The elements of this matrix could be further aggregated and averaged based on the category to which events were assigned, reducing to an asymmetric, matrix (rows and columns correspond to the three event types).
This first step was repeated for all pairs of subjects, each time yielding an analogous matrix. At its completion this, second step yielded a similarity matrix, whose rows and columns could be reordered by event types. We show this matrix in Fig. 2d.
5.4.7 Co-fluctuation matrix differences
In the main text, we described differences in the co-fluctuation matrices associated with different event types. To assess these differences, we performed the following set of calculations. For a given pair of event types, we identified all instances of each type and calculated, at the level of edges, the differences in their co-fluctuation magnitude and the mean sign of those differences. We then retained the sign of these differences. The result is a matrix for each subject whose elements are . Note that this calculation was performed within subjects.
Next, we examined whether, across subjects, edges exhibited concordant behavior using a paired-samples t-test. That is, we assessed whether, across subjects, an edge’s co-fluctuation magnitude under one event type was consistently greater than or less than that of another event type. This procedure was repeated for every pair of event types.
Data and Code Availability
Code for event detection as well as the detection of synchronous events can be found at: https://github.com/JacobColbyTanner/find-synchronous-events. The main data used in this project are already publicly available. For more information on how to access, visit: db.humanconnectome.org. The data collected at Indiana University are available upon request.
Author Contributions
J.C.T., O.S., and R.F.B. conceived of experiments, designed analyses, wrote the initial versions of the manuscript, and edited the manuscript. J.C.T. performed all analyses. J.F., L.B., and D.P.K. contributed data. All authors helped edit the manuscript.
Declaration of Competing Interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
Data were provided [in part] by the Human Connectome Project, WU-Minn Consortium (Principal Investigators: David Van Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University. This material is based on work supported by NSF Grant IIS-2023985 (R.F.B., O.S.), NSF Grant 1735095, Interdisciplinary Training in Complex Networks (J.C.T.), and Indiana University Office of the Vice President for Research Emerging Area of Research Initiative, Learning: Brains, Machines and Children (R.F.B.). This research was supported in part by Lilly Endowment, Inc., through its support for the Indiana University Pervasive Technology Institute.
Supplementary Materials
Supplementary material for this article is available with the online version here: https://doi.org/10.1162/imag_a_00026.