Medial Orbitofrontal Cortex, Dorsolateral Prefrontal Cortex, and Hippocampus Differentially Represent the Event Saliency

“I was waiting at home for my friend. I made some tea, washed the cups, and poured hot water. Then I felt everything shaking. It was an earthquake. I put the cup down and waited to see if there was an aftershock. Just about then, my friend arrived.” We experience the world as a sequence of events, but we remember them as segmented sequences. During encoding, perceiving unusual events separates the flow of events, which is an event segmentation process (Zacks, Speer, Swallow, Braver, & Reynolds, 2007; Zacks & Swallow, 2007), so that each segment contains the relationship of events that occurred in a similar circumstance (Zwaan & Radvansky, 1998). In this process, sequences of events are organized to create a hierarchical relationship (Kurby & Zacks, 2008; Zacks & Swallow, 2007; Zwaan & Radvansky, 1998), where a sequence of less notable events are embedded in an overriding structure of segmented sequences (Hanson & Hirst, 1989). In this example, making some tea was embedded in a friend's visit. We hypothesized that a basic model to support the hierarchical relationship of sequences relies on the magnitude of event deviancy. In principle, the hierarchical relationship can be reconstructed from event saliency where less salient events are more temporally associated with prior events than more salient new events (Figure 1C; also see Yeung, Yeo, & Liu, 1996).

The magnitude of deviance of events is referred to as “event saliency” and is quantified by crowdsourcing. Previously, event saliency has been used to determine the probability of a deviant event in a linguistic experiment (Coulson, King, & Kutas, 1998), but recently, with the availability of videos and advances in computer vision, this term is used for quantifying the magnitude of deviation in a flow of a movie. Event saliency is not determined by changes in a visual scene but relies on following the movies and noting significant changes in the flow of events (Zhang, Han, Jiang, Ye, & Chang, 2017; Rosani, Boato, & De Natale, 2015). Accordingly, event saliency is measured by either asking an independent group to identify the boundaries or, in linguistics, studying the transition from one word to another, which is extracted from a large body of literature.

Both event association and deviancy detection are linked to the hippocampus, OFC, and dorsolateral prefrontal cortex (PFC; Paz et al., 2010; Zacks et al., 2007; Nobre, Coull, Frith, & Mesulam, 1999; Knight, 1996). We reasoned deviant detection reflects event saliency; this signal would also reflect a temporal association of events. Here, we tested the prediction that high-frequency neural activity in subregions of PFC and the hippocampus tracked event saliency.

PFC–hippocampal neural network is also engaged in prospective coding (Jafarpour, Piai, Lin, & Knight, 2017; Brown et al., 2016; Hindy, Ng, & Turk-Browne, 2016; Hsieh & Ranganath, 2015; Hsieh, Gruber, Jenkins, & Ranganath, 2014), which enhances event segmentation (Schütz-Bosbach & Prinz, 2007). Note that anticipated salient events are different from novel events. For example, a salient event like a friend's planned visit is anticipated, whereas a salient event such as an earthquake is novel. Here, we predicted that the neural representation of event saliency would be different for novel and predictable salient events.

Generating sequences of novel events in an experimental condition is challenging (Zarcone, van Schijndel, Vogels, & Demberg, 2016). Previous studies have used a discrete experimental design, comparing the neural activity at the time of perceiving deviant events, that is, event boundaries (Kurby & Zacks, 2008), to the neural activity at the time of perceiving nonboundary events (Whitney et al., 2009; Speer, Zacks, & Reynolds, 2007; Zacks et al., 2007). However, after encountering the first few deviant events, participants anticipate the new events; thus, the later deviant events are no longer novel; instead, they become anticipated salient events. A flow of events, such as observed in a movie, has numerous event boundaries, with a range of anticipated or novel saliency providing an ideal experimental design to address this issue.

We recorded local field potential using intracranial electroencephalography (iEEG) from 10 patients with epilepsy who had electrodes implemented for clinical purposes. Patients passively watched a movie (Figure 1). The analysis focused on the local activity captured as the power in the high-frequency activity (HFA; 50–150 Hz) that serves as a metric for local neural activation (Rich & Wallis, 2017; Lachaux, Axmacher, Mormann, Halgren, & Crone, 2012; Jacobs & Kahana, 2009; Belitski et al., 2008; Ray, Crone, Niebur, Franaszczuk, & Hsiao, 2008).

The study protocol was approved by the Office for the Protection of Human Subjects of the University of California, Berkeley, and the University of California, Irvine. All participants provided written informed consent before participating.

Ten epileptic patients who had stereotactically implanted depth electrodes to localize the seizure onset zone for subsequent surgical resection participated in this study (four women, mean age = 37 years, SD = 11, age range = 22–58 years; Table 1). The electrodes were placed at the University of California, Irvine, Medical Center, with 5-mm interelectrode spacing. All patients had normal (or corrected) vision. No seizure occurred during task administration. Two independent neurologists inspected the neural activity and identified the electrodes with an epileptic activity, which were excluded from the analysis so that all electrodes included in the analysis were nonpathological and free of epileptogenic spikes. Any segment where focal spikes spread to other brain regions were also excluded from the analysis (Table 1). Electrode coverage included the medial-temporal lobe and the PFC, depending on their clinical requirements (Figure 2). Electrodes were localized in the patient's native space and then transferred to MNI space for visualizing the group coverage. We studied electrodes in three ROIs: the lateral PFC, the OFC, and the hippocampus (Table 1 and Figure 2).

A control group of 80 healthy adults participated in a movie segmentation test (53 women, mean age = 28 years, SD = 14, age range = 18–68 years). Thirty-one participants were 18–21 years old, 29 participants were 21–30 years old, and 20 participants were older than 30 years old.

Salient events can occur frequently in a flow of events, such as serving customers in a busy café, or they can be infrequent, such as driving along a desert highway. Participants watched a short mute animation (∼3 min long) that had frequent salient events. The movie was a short version of the animation designed by Ali Derakhshi, named “Wildlife” or “Hayat-e Vahsh,” the episode on lions. The movie was selected so that participants have not watched it before, the visual angle was kept similar (Figure 1A), and events with various magnitudes of saliency occurred in a short period. Critically, the storyline of the movie changed so we could test the effect of novel versus anticipated salient events (link to the movie: https://www.youtube.com/watch?v=Q_guH9vA0sk).

The movie had an overarching cliché love triangle story. It starts by showing a few animal couples going back and forth in a park (this part gets repetitive after two repetitions). Then, there is a small lion that looks heartbroken. The lion sees a lioness, but there is a bigger lion that also wants to meet the lioness. The two lions fight for the lioness's attention through a series of matches. After each match, the score is shown on board (this part is repetitive and predictable). However, the score is not immediately shown after an eating contest. After the bigger lion wins the eating contest, it eats the small lion's food too. Then, the score is shown, and the matches continue. The small lion loses the competition and moves on. The end of the movie shows that the small lion meets a lioness again (this part was repetitive). This movie had periods with an anticipated flow. For example, after each game that the bigger lion wins, the scoreboard is shown.

The iEEG group passively watched the muted movie, but the behavioral control group watched the muted movie and concurrently segmented the movies into episodes. We instructed the behavioral group to press a key “whenever something new happened.” We clarified that “

We recorded the timing of keypresses during segmentation and target detection tasks. Participants' RT during the target detection task was measured as the difference between the target onsets and the responses. We excluded consecutive keypresses for segmentation if the interval was less than 100 msec (double key registration). The averaged RT per participant was subtracted from the timing of keypresses for movie segmentation to normalize the timing of event boundaries. The number of segmented events was accumulated across participants at 1.5-sec epochs. The number of events in the 1.5-sec epochs reflected the saliency of events (see also Ben-Yakov & Henson, 2018). Accordingly, an event was more salient if more people reported it as an event boundary, and an event was considered less salient if fewer people marked them as an event boundary (see Figure 1B for the range of saliency scale). The autocorrelation analysis of the event saliency was tested using the Ljung–Box test that was implemented in R (R Development Core Team, 2014). There was no significant autocorrelation in the saliency magnitude in 1.5-sec epochs (χ² = 1.922, df = 1, p = .1656). Likewise, autocorrelation in the epochs of 1.5-sec HFAs was negligible, allowing to using permutation test and applying the event saliency magnitude for statistical analysis as outlined below.

A hierarchical clustering of event relationships (Figure 1) was constructed by applying a binary hierarchical clustering algorithm in R (R Development Core Team, 2014) using distances between events. The distance between two events was measured by adding the saliency magnitude of all the events between the two events. Accordingly, the events that had many highly salient events in between them had larger distance than the events with less salient events between them.

iEEG data were acquired using the Nihon Kohden recording system, analog-filtered above 0.01 Hz and digitally sampled at 5 kHz or 10 kHz. A photodiode recorded the luminance of a corner of the screen to track the timing of the movie presentation. Two independent neurologists selected the electrodes that showed both epileptic activities and epochs with seizure spread. Only electrodes in nonpathological regions were included in the analysis.

All EEG analyses were run in R, MATLAB 2015a, and Fieldtrip toolbox (Oostenveld, Fries, Maris, & Schoffelen, 2011) offline. We applied a 2-Hz-wide stopband Butterworth notch filter at 60-Hz line power noise and harmonics and then down-sampled the data to 1 kHz using resample() MATLAB function via Fieldtrip. The function applies an antialiasing finite impulse response lowpass filter and compensates for the delay introduced by the filter. All electrodes were re-referenced to a neighboring electrode (i.e., bipolar reference). The continuous signal was then cropped in 1.5-sec-long epochs with no overlaps. The epochs were bandpass filtered for HFA (50–150 Hz) using padding and a Hamming window. The Hilbert transformation was applied to the filtered data for extracting the power.

The correlation between event saliency and HFA in each electrode was calculated using the Spearman correlation. For estimating the p value, we used a nonparametric statistical permutation test because the nonoverlapping 1.5-sec epochs of HFA were interchangeable. Note that the magnitudes of saliency and HFA epochs were not significantly autocorrelated. The null distribution was made from 1000 iterations of surrogated trial labels. In each iteration, the maximum correlation between HFA and saliency magnitude was taken across all electrodes in an ROI (namely, the lateral PFC, the OFC, and the hippocampus) for each participant. The proportion of HFA–saliency magnitude correlation coefficients in the null distribution that was more than the observed correlation coefficient yielded the nonparametric corrected p value for the observed correlation. A p value of <.05 was considered significant.

We tested if HFA–saliency magnitude correlation changed with anticipating salient events using a linear mixed-effects model. We recalculated the correlations between HFA and saliency magnitudes in each brain region in sliding windows of 15 sec with overlaps of 7.5 sec (24 bins). We used a linear mixed-effects model to test the effect of novel (n = 13) or anticipated periods of salient events (n = 11; rep = 0 for novel and 1 for repetitive storylines; Figure 4) from the three ROIs (the correlation coefficients [R] in each electrode region; the hippocampus, OFC, and dorsolateral PFC). We used the linear mixed-effects model in MATLAB (fitlme) to account for the different number of electrodes in each ROI of a subject and the nested effect of time: formulated as R ∼ rep + (rep | subject : electrodes) + (1 | rep : time_bin). ANOVA was applied for the result of F tests for the fixed-effect term in the linear mixed-effects model.

Electrode locations were reconstructed and visualized in MATLAB using the Fieldtrip toolbox (Stolk et al., 2018). We manually selected electrodes on the postimplantation CT, which was coregistered to the preimplantation MRI using SPM (Ashburner & Friston, 1997) to maximize the accuracy of the reconstructions. A neurologist identified the electrodes' locations. We then normalized each patient's preimplantation MRI to the MNI-152 template brain using SPM to obtain the electrode positions in MNI space (Ashburner & Friston, 1999). If electrode locations in MNI space did not correspond to electrode locations in native (participant) space after normalization (e.g., an electrode is within hippocampus in native space but appears outside the hippocampus in MNI space after normalization), then electrode locations were manually adjusted to represent their true locations in native space. Electrode locations for bipolar re-referenced channels were calculated as the midpoint between the two electrodes (Burke et al., 2013, 2014; Long, Burke, & Kahana, 2014). Representations of the cerebral cortex were generated using FreeSurfer (Dale, Fischl, & Sereno, 1999) and representations of the hippocampus were generated from the Desikan–Killiany atlas (Desikan et al., 2006) using Fieldtrip. Brodmann's areas were inferred from Bioimage Suite package (bioimagesuite.yale.edu/).

We identified the magnitude of salient events in the movie by studying a separate group of adults (n = 80). This group indicated when, during the movie, a new episode started (i.e., perceiving an event boundary). The metric of event saliency magnitude was the proportion of identified event boundaries in a short time window of the movie (about 20 frames or 1.5 sec; Figure 1B; total movie time was 3 min). Epochs of 1.5 sec resulted in saliency magnitude of 0–0.6 (1 would be the maximum saliency magnitude when every participant agrees that, during the same 1.5 sec, an event boundary occurred). A distance matrix was constructed from the sum of the saliency of events that occurred between pairs of events and was used for binary hierarchical event clustering (see the Methods section; Figure 1). We tested the hypothesis that the magnitude of event saliency was tracked in the targeted regions. Ranked (Spearman) correlation and nonparametric permutation tests for statistical results were applied. The interchangeable nonoverlapping HFA in 1.5-sec epochs allowed using nonparametric permutation testing. Cluster-corrected p values are reported.

We observed that HFA correlated with the magnitude of event saliency was captured by the behavior of the independent rating group. The neural effect was clustered in dorsolateral PFC (BA 6, BA 8, BA 9, and BA 10; in five of seven patients; seven of nine patients with lateral PFC electrodes had dorsolateral PFC coverage). The effect was also detected in the hippocampus (BA 54; in three of three patients) and the medial OFC (BA 11; in three of three patients with medial OFC coverage; three of seven patients with OFC electrodes had medial PFC coverage; Figure 3). See Figure 3 for the correlation coefficient of all electrodes and Table 2 for statistical results of electrodes that showed a significant correlation (the R value of electrodes with p > .05 is color-coded in Figure 3).

We conducted a planned analysis on the electrodes that showed a significant correlation (i.e., task relevant) to assess the effects of anticipation (Table 2). We recalculated the correlation coefficient between HFA and event saliency magnitude in 15-sec-long sliding windows (7.5-sec overlaps), resulting in 24 tested windows, of which 11 had repetitive storylines and 13 were novel. The flow of salient events in 46% of the sliding windows was anticipated. A storyline was predictable if the same type of event reoccurred more than twice, such as repetition of animals going back and forth or repetition of scoring in a competition. The results of a linear mixed-effect model showed that the correlation coefficient between HFA and saliency magnitude was higher in the OFC when the salient events were anticipated than when they were novel (Figure 4A; OFC, F(1, 142) = 4.3, p = .039), and this effect was observed in all patients with task-relevant electrodes (Figure 4B). There was no difference between novel and anticipated salient events in the hippocampus, F(1, 166) = 0.39, p = .52, or the dorsolateral PFC, F(1, 526) = 1.26, p = .26.

During encoding, a sequence of events is segmented to construct a hierarchical representation of event associations (Kurby & Zacks, 2008; Zacks & Swallow, 2007; Zwaan & Radvansky, 1998), with clusters of associated events represented in lower levels of a hierarchy and the associations of the clusters of events represented in higher levels of the hierarchy. The construction of such a hierarchical association requires linking relevant events and separating events that occur in different circumstances. For instance, a circumstance changes with perceiving a deviant event. Detecting the magnitude of event saliency can also contribute to establishing the structure of associations. When the newly perceived event is not salient, the event is closely associated with the preceding events; however, if the new event is highly salient, it should be separated from the preceding events. Here, we report distributed neural regions that detect the magnitude of deviance (i.e., event saliency) in a flow of events, including dorsolateral PFC and hippocampus, and further show that anticipating the deviant events affects the OFC activities.

We used event segmentations of a large control population (behavior group) who watched the silent movie to infer the event segmentation in another group with intracranial electrodes (iEEG group) who watched the same movie. The behavior group's event segmentation provided the event saliency of the entire movie. The RT of each participant in this group was estimated from a target detection task and used for normalizing the timing of event boundaries (see the Methods section). We inferred the saliency from the proportion of people that reported an event boundary in each movie epoch. The 1.5-sec windows provided interchangeable epochs of data for using correlation and permutation tests (see the Methods section).

The iEEG group passively watched a movie and did not know about the segmentation task, allowing us to study spontaneous and naturalistic neural processing during parsing a continuous flow of events. We observed that the HFA that is linked to nearby single neural activity (Rich & Wallis, 2017; Lachaux et al., 2012; Jacobs & Kahana, 2009; Belitski et al., 2008; Ray, Crone, et al., 2008) increased proportionally with event saliency in the hippocampus, dorsolateral PFC, and medial OFC.

HFA in the dorsolateral PFC in the iEEG group tracked event saliency magnitude. Dorsolateral PFC is critical for guiding attention (Corbetta & Shulman, 2002; Hopfinger, Buonocore, & Mangun, 2000; Kastner, Pinsk, De Weerd, Desimone, & Ungerleider, 1999; Paus, 1996), and increased HFA may, in part, be due to attention to novel events (Ray, Niebur, Hsiao, Sinai, & Crone, 2008; Zacks et al., 2001). Dorsolateral PFC is also engaged in cognitive control and conflict monitoring (Miller & Cohen, 2001; MacDonald, Cohen, Stenger, & Carter, 2000) by detecting new associations of categories and exemplars (Dolan & Fletcher, 1997). Accordingly, the observed additional correlation between HFA and event saliency in dorsolateral PFC reflects the demand for event segmentation and updating the event circumstance (Reynolds, Zacks, & Braver, 2007; Zacks et al., 2007).

Event saliency also correlated with HFA in the hippocampus. Hippocampal activity has been linked to the representation of event associations (Mack, Love, & Preston, 2017; Quiroga, 2012; Quiroga, Reddy, Kreiman, Koch, & Fried, 2005; Ekstrom et al., 2003). The hippocampal representation changes with salient changes in the environment (Shapiro, Tanila, & Eichenbaum, 1997), and its activity increases with detecting salient events (Chen, Cook, & Wagner, 2015; Chen et al., 2013; Axmacher et al., 2010; Kumaran & Maguire, 2007; Wittmann, Bunzeck, Dolan, & Düzel, 2007; Lisman & Otmakhova, 2001; Knight, 1996). Recent studies showed that hippocampal representations reflect the scale of topological saliencies of an environment, such as changes in the spatial closeness of streets or the centrality of the streets (Javadi et al., 2017), and the scale of deviance from expectation (Chen et al., 2015). Also, the hippocampal BOLD signal tracked the saliency of event boundaries when people watched movies (Ben-Yakov & Henson, 2018). Here, we propose that small deviance induces only a minor change in the hippocampal representation so that close events share more similar hippocampus representations than far events (Ezzyat & Davachi, 2014). These results expand the pattern separation mechanism for distinguishing similar visual associations attributed to the hippocampus (Yassa & Stark, 2011) to a mechanism for identifying the scale of event separation. Pattern separation for visual stimuli engages dentate gyrus in the hippocampus (Baker et al., 2016; Berron et al., 2016), but what the subregion of the human hippocampus contributes to the deviant detection is unknown (see Lisman & Grace, 2005, for the novelty signal in the rodent's subiculum and Knierim & Neunuebel, 2016, for mismatch signal in subregions of rodent's hippocampus).

We also observed a similar saliency magnitude effect in the medial OFC (BA 11 but not in the lateral OFC), with increased HFA for highly salient events. This observation is akin to the representation of saliency in the nonhuman primates' OFC, captured by HFA (Rich & Wallis, 2016, 2017). In humans, breaching expectations increases the OFC activity (Mikutta et al., 2015; Duarte, Henson, Knight, Emery, & Graham, 2009; Nobre et al., 1999). OFC also represents the saliency of anticipated events (Metereau & Dreher, 2015; Bechara, Tranel, Damasio, & Damasio, 1996). The reflection of the saliency magnitude suggests that OFC represents the structure of the event association. Notably, anticipation is a critical feature for encoding sequences of events because the gist of a previous experience shapes the expected context (Reynolds et al., 2007; Purcell, 1986). Here, the HFA in medial OFC tracked the event saliency better when the salient events were anticipated than during a novel intrusion of events, suggesting that the anticipated structure of event associations is represented in the medial OFC.

An important question concerns the dynamics of interaction in the neural network for representing the structure of event associations. For example, disturbing the input from the hippocampus to OFC impairs representing task structures in rodents (Wikenheiser, Marrero-Garcia, & Schoenbaum, 2017). It is not clear whether hippocampal deviancy detection is essential for constructing the OFC signal in humans or whether other brain regions such as midbrain structures contribute to detecting the magnitude of deviance (Dürschmid et al., 2016; Wittmann et al., 2007). For instance, hippocampal response to deviant events is associated with activity in the substantia nigra and ventral tegmental area (Murty & Adcock, 2014; Wittmann et al., 2007). It is also suggested that the hippocampus provides a novelty signal to the nucleus accumbens (Dürschmid et al., 2016). Although hippocampus activity is linked to anticipation (Jafarpour et al., 2017; Hindy et al., 2016; Hsieh et al., 2014), an outstanding question is whether the prediction is made by the hippocampus or is under control of other brain regions, such as the PFC. Systematically, comparison of regional activity, however, requires simultaneous recordings from both regions in a patient. A caveat of this iEEG study is that all the patients did not have sufficient coverage from multiple task-relevant regions to definitively study the dynamics of the network.

Event segmentation requires tracing the association of a new event to the preceding events (Zwaan & Radvansky, 1998). Detecting a new event's deviance magnitude helps with an accurate association of the event to the preceding sequence. Accordingly, segmented sequences that are separated by small surprises are more associated in comparison to a sequence separated by a big surprise. Detecting deviant events is known to increase neural activity in the PFC and the hippocampus (Long, Lee, & Kuhl, 2016; Axmacher et al., 2010; Bunzeck, Dayan, Dolan, & Duzel, 2010; Kumaran & Maguire, 2007; Strange, Duggins, Penny, Dolan, & Friston, 2005; Knight, 1996). Both brain regions, albeit differently, represent the associations in an experimental setup (Wikenheiser & Schoenbaum, 2016; Wilson, Takahashi, Schoenbaum, & Niv, 2014; O'Keefe & Nadel, 1978; Tolman, 1948). Here, we showed that the hippocampus and PFC regions tracked the scale of event saliency in a movie, and in the medial OFC, this effect is stronger when the salient events were anticipated than for novel events. We propose that a core function of the hippocampus, dorsolateral PFC, and medial OFC network is to construct the event association structure, akin to a task structure (Wikenheiser & Schoenbaum, 2016; Wilson et al., 2014).

This research used statistical consulting resources provided by the Center for Statistics and the Social Sciences, University of Washington. This work was sponsored by the James S. McDonnell Foundation, National Institute of Neurological Disorders grant R37 NS21135 (to R. T. K.), and the University of California, Irvine School of Medicine Bridge Fund (to J. J. L.). The authors are indebted to the patients for their participation. We thank Jie Zheng and members of the Knightlab for helping with data collection. We also thank Prof. Elizabeth Buffalo for helpful discussions.

Reprint requests should be sent to Anna Jafarpour, Department of Physiology and Biophysics, University of Washington, 1705 NE Pacific St., HSB Box 357290, Seattle, WA 98195, or via e-mail: [email protected].