Abstract

Two primary functions attributed to the hippocampus and prefrontal cortex (PFC) network are retaining the temporal and spatial associations of events and detecting deviant events. It is unclear, however, how these two functions converge into one mechanism. Here, we tested whether increased activity with perceiving salient events is a deviant detection signal or contains information about the event associations by reflecting the magnitude of deviance (i.e., event saliency). We also tested how the deviant detection signal is affected by the degree of anticipation. We studied regional neural activity when people watched a movie that had varying saliency of a novel or an anticipated flow of salient events. Using intracranial electroencephalography from 10 patients, we observed that high-frequency activity (50–150 Hz) in the hippocampus, dorsolateral PFC, and medial OFC tracked event saliency. We also observed that medial OFC activity was stronger when the salient events were anticipated than when they were novel. These results suggest that dorsolateral PFC and medial OFC, as well as the hippocampus, signify the saliency magnitude of events, reflecting the hierarchical structure of event associations.

INTRODUCTION

“I was waiting at home for my friend. I made some tea, washed the cups, and poured hot water. Then I felt everything shaking. It was an earthquake. I put the cup down and waited to see if there was an aftershock. Just about then, my friend arrived.” We experience the world as a sequence of events, but we remember them as segmented sequences. During encoding, perceiving unusual events separates the flow of events, which is an event segmentation process (Zacks, Speer, Swallow, Braver, & Reynolds, 2007; Zacks & Swallow, 2007), so that each segment contains the relationship of events that occurred in a similar circumstance (Zwaan & Radvansky, 1998). In this process, sequences of events are organized to create a hierarchical relationship (Kurby & Zacks, 2008; Zacks & Swallow, 2007; Zwaan & Radvansky, 1998), where a sequence of less notable events are embedded in an overriding structure of segmented sequences (Hanson & Hirst, 1989). In this example, making some tea was embedded in a friend's visit. We hypothesized that a basic model to support the hierarchical relationship of sequences relies on the magnitude of event deviancy. In principle, the hierarchical relationship can be reconstructed from event saliency where less salient events are more temporally associated with prior events than more salient new events (Figure 1C; also see Yeung, Yeo, & Liu, 1996).

Figure 1. 

A hierarchical structure of the movie can be extracted from event saliency. (A) Patients passively watched a (∼3 min) muted animation that they did not see before. The movie had a mixture of novel and anticipated new events. (B) We chunked the movie into 129 interchangeable epochs of 1.5 sec. The epochs had a range of event saliency, defined as the proportion of an independent group of participants (n = 80) that determined event boundaries in the movie epochs. (C) Event saliency was used to construct the hierarchical relationship of epochs, with distance being the sum of event saliency between epochs. The heat map shows the sum of saliency of the events that occurred between each pair of events (ranged between 0 and 18.8, which is the largest sum of saliency of a pair of events). The row and column order have been reordered based on the hierarchical clustering results. All the epochs are displayed in rows and columns (every third column is numbered in the illustration), and temporally adjacent events are next to each other. The right and left branches of the hierarchy do not imply any order and can be flipped because the plot is symmetric. (D) Zoomed in view of the first 10.5 sec of the movie (marked branches in C). The graph shows the structure of event associations at the start of the movie.

Figure 1. 

A hierarchical structure of the movie can be extracted from event saliency. (A) Patients passively watched a (∼3 min) muted animation that they did not see before. The movie had a mixture of novel and anticipated new events. (B) We chunked the movie into 129 interchangeable epochs of 1.5 sec. The epochs had a range of event saliency, defined as the proportion of an independent group of participants (n = 80) that determined event boundaries in the movie epochs. (C) Event saliency was used to construct the hierarchical relationship of epochs, with distance being the sum of event saliency between epochs. The heat map shows the sum of saliency of the events that occurred between each pair of events (ranged between 0 and 18.8, which is the largest sum of saliency of a pair of events). The row and column order have been reordered based on the hierarchical clustering results. All the epochs are displayed in rows and columns (every third column is numbered in the illustration), and temporally adjacent events are next to each other. The right and left branches of the hierarchy do not imply any order and can be flipped because the plot is symmetric. (D) Zoomed in view of the first 10.5 sec of the movie (marked branches in C). The graph shows the structure of event associations at the start of the movie.

The magnitude of deviance of events is referred to as “event saliency” and is quantified by crowdsourcing. Previously, event saliency has been used to determine the probability of a deviant event in a linguistic experiment (Coulson, King, & Kutas, 1998), but recently, with the availability of videos and advances in computer vision, this term is used for quantifying the magnitude of deviation in a flow of a movie. Event saliency is not determined by changes in a visual scene but relies on following the movies and noting significant changes in the flow of events (Zhang, Han, Jiang, Ye, & Chang, 2017; Rosani, Boato, & De Natale, 2015). Accordingly, event saliency is measured by either asking an independent group to identify the boundaries or, in linguistics, studying the transition from one word to another, which is extracted from a large body of literature.

Both event association and deviancy detection are linked to the hippocampus, OFC, and dorsolateral prefrontal cortex (PFC; Paz et al., 2010; Zacks et al., 2007; Nobre, Coull, Frith, & Mesulam, 1999; Knight, 1996). We reasoned deviant detection reflects event saliency; this signal would also reflect a temporal association of events. Here, we tested the prediction that high-frequency neural activity in subregions of PFC and the hippocampus tracked event saliency.

PFC–hippocampal neural network is also engaged in prospective coding (Jafarpour, Piai, Lin, & Knight, 2017; Brown et al., 2016; Hindy, Ng, & Turk-Browne, 2016; Hsieh & Ranganath, 2015; Hsieh, Gruber, Jenkins, & Ranganath, 2014), which enhances event segmentation (Schütz-Bosbach & Prinz, 2007). Note that anticipated salient events are different from novel events. For example, a salient event like a friend's planned visit is anticipated, whereas a salient event such as an earthquake is novel. Here, we predicted that the neural representation of event saliency would be different for novel and predictable salient events.

Generating sequences of novel events in an experimental condition is challenging (Zarcone, van Schijndel, Vogels, & Demberg, 2016). Previous studies have used a discrete experimental design, comparing the neural activity at the time of perceiving deviant events, that is, event boundaries (Kurby & Zacks, 2008), to the neural activity at the time of perceiving nonboundary events (Whitney et al., 2009; Speer, Zacks, & Reynolds, 2007; Zacks et al., 2007). However, after encountering the first few deviant events, participants anticipate the new events; thus, the later deviant events are no longer novel; instead, they become anticipated salient events. A flow of events, such as observed in a movie, has numerous event boundaries, with a range of anticipated or novel saliency providing an ideal experimental design to address this issue.

We recorded local field potential using intracranial electroencephalography (iEEG) from 10 patients with epilepsy who had electrodes implemented for clinical purposes. Patients passively watched a movie (Figure 1). The analysis focused on the local activity captured as the power in the high-frequency activity (HFA; 50–150 Hz) that serves as a metric for local neural activation (Rich & Wallis, 2017; Lachaux, Axmacher, Mormann, Halgren, & Crone, 2012; Jacobs & Kahana, 2009; Belitski et al., 2008; Ray, Crone, Niebur, Franaszczuk, & Hsiao, 2008).

METHODS

Approval

The study protocol was approved by the Office for the Protection of Human Subjects of the University of California, Berkeley, and the University of California, Irvine. All participants provided written informed consent before participating.

Participants

Intracranial EEG

Ten epileptic patients who had stereotactically implanted depth electrodes to localize the seizure onset zone for subsequent surgical resection participated in this study (four women, mean age = 37 years, SD = 11, age range = 22–58 years; Table 1). The electrodes were placed at the University of California, Irvine, Medical Center, with 5-mm interelectrode spacing. All patients had normal (or corrected) vision. No seizure occurred during task administration. Two independent neurologists inspected the neural activity and identified the electrodes with an epileptic activity, which were excluded from the analysis so that all electrodes included in the analysis were nonpathological and free of epileptogenic spikes. Any segment where focal spikes spread to other brain regions were also excluded from the analysis (Table 1). Electrode coverage included the medial-temporal lobe and the PFC, depending on their clinical requirements (Figure 2). Electrodes were localized in the patient's native space and then transferred to MNI space for visualizing the group coverage. We studied electrodes in three ROIs: the lateral PFC, the OFC, and the hippocampus (Table 1 and Figure 2).

Table 1. 
Patient Electrode Coverage
PatientAgeSexDominant HandOFCHippocampusLateral PFCNo. of trials
50 Male Right 10 15 112 
46 Male Right   17 125 
34 Female Right     104 
31 Male Left   15 110 
40 Female Left 11 20 129 
58 Female Right 20 123 
34 Male Right     129 
22 Female Right 10   16 122 
33 Male Right     10 126 
10 23 Male Right     33 129 
PatientAgeSexDominant HandOFCHippocampusLateral PFCNo. of trials
50 Male Right 10 15 112 
46 Male Right   17 125 
34 Female Right     104 
31 Male Left   15 110 
40 Female Left 11 20 129 
58 Female Right 20 123 
34 Male Right     129 
22 Female Right 10   16 122 
33 Male Right     10 126 
10 23 Male Right     33 129 

This table contains patients' age (in years); sex (male or female); dominant hand (right or left); the number of bipolar-referenced electrodes in any region of the OFC, hippocampus, or lateral PFC; and the number of trials (out of 129 total trials).

Figure 2. 

Patients' electrode coverage is color-coded by the patients' number. (A–D) shows PFC and OFC coverage from (A) the right sagittal, (B) the left sagittal, (C) the coronal, and (D) the inferior view. (E) Hippocampal coverage in a 3-D glass hippocampus from the superior anterior view.

Figure 2. 

Patients' electrode coverage is color-coded by the patients' number. (A–D) shows PFC and OFC coverage from (A) the right sagittal, (B) the left sagittal, (C) the coronal, and (D) the inferior view. (E) Hippocampal coverage in a 3-D glass hippocampus from the superior anterior view.

Behavioral Experiment

A control group of 80 healthy adults participated in a movie segmentation test (53 women, mean age = 28 years, SD = 14, age range = 18–68 years). Thirty-one participants were 18–21 years old, 29 participants were 21–30 years old, and 20 participants were older than 30 years old.

Experimental Design

Salient events can occur frequently in a flow of events, such as serving customers in a busy café, or they can be infrequent, such as driving along a desert highway. Participants watched a short mute animation (∼3 min long) that had frequent salient events. The movie was a short version of the animation designed by Ali Derakhshi, named “Wildlife” or “Hayat-e Vahsh,” the episode on lions. The movie was selected so that participants have not watched it before, the visual angle was kept similar (Figure 1A), and events with various magnitudes of saliency occurred in a short period. Critically, the storyline of the movie changed so we could test the effect of novel versus anticipated salient events (link to the movie: https://www.youtube.com/watch?v=Q_guH9vA0sk).

The movie had an overarching cliché love triangle story. It starts by showing a few animal couples going back and forth in a park (this part gets repetitive after two repetitions). Then, there is a small lion that looks heartbroken. The lion sees a lioness, but there is a bigger lion that also wants to meet the lioness. The two lions fight for the lioness's attention through a series of matches. After each match, the score is shown on board (this part is repetitive and predictable). However, the score is not immediately shown after an eating contest. After the bigger lion wins the eating contest, it eats the small lion's food too. Then, the score is shown, and the matches continue. The small lion loses the competition and moves on. The end of the movie shows that the small lion meets a lioness again (this part was repetitive). This movie had periods with an anticipated flow. For example, after each game that the bigger lion wins, the scoreboard is shown.

The iEEG group passively watched the muted movie, but the behavioral control group watched the muted movie and concurrently segmented the movies into episodes. We instructed the behavioral group to press a key “whenever something new happened.” We clarified that “

Behavioral Analysis

We recorded the timing of keypresses during segmentation and target detection tasks. Participants' RT during the target detection task was measured as the difference between the target onsets and the responses. We excluded consecutive keypresses for segmentation if the interval was less than 100 msec (double key registration). The averaged RT per participant was subtracted from the timing of keypresses for movie segmentation to normalize the timing of event boundaries. The number of segmented events was accumulated across participants at 1.5-sec epochs. The number of events in the 1.5-sec epochs reflected the saliency of events (see also Ben-Yakov & Henson, 2018). Accordingly, an event was more salient if more people reported it as an event boundary, and an event was considered less salient if fewer people marked them as an event boundary (see Figure 1B for the range of saliency scale). The autocorrelation analysis of the event saliency was tested using the Ljung–Box test that was implemented in R (R Development Core Team, 2014). There was no significant autocorrelation in the saliency magnitude in 1.5-sec epochs (χ2 = 1.922, df = 1, p = .1656). Likewise, autocorrelation in the epochs of 1.5-sec HFAs was negligible, allowing to using permutation test and applying the event saliency magnitude for statistical analysis as outlined below.

A hierarchical clustering of event relationships (Figure 1) was constructed by applying a binary hierarchical clustering algorithm in R (R Development Core Team, 2014) using distances between events. The distance between two events was measured by adding the saliency magnitude of all the events between the two events. Accordingly, the events that had many highly salient events in between them had larger distance than the events with less salient events between them.

iEEG Data Collection and Preprocessing

iEEG data were acquired using the Nihon Kohden recording system, analog-filtered above 0.01 Hz and digitally sampled at 5 kHz or 10 kHz. A photodiode recorded the luminance of a corner of the screen to track the timing of the movie presentation. Two independent neurologists selected the electrodes that showed both epileptic activities and epochs with seizure spread. Only electrodes in nonpathological regions were included in the analysis.

All EEG analyses were run in R, MATLAB 2015a, and Fieldtrip toolbox (Oostenveld, Fries, Maris, & Schoffelen, 2011) offline. We applied a 2-Hz-wide stopband Butterworth notch filter at 60-Hz line power noise and harmonics and then down-sampled the data to 1 kHz using resample() MATLAB function via Fieldtrip. The function applies an antialiasing finite impulse response lowpass filter and compensates for the delay introduced by the filter. All electrodes were re-referenced to a neighboring electrode (i.e., bipolar reference). The continuous signal was then cropped in 1.5-sec-long epochs with no overlaps. The epochs were bandpass filtered for HFA (50–150 Hz) using padding and a Hamming window. The Hilbert transformation was applied to the filtered data for extracting the power.

Correlation Analysis between HFA and Event Saliency

The correlation between event saliency and HFA in each electrode was calculated using the Spearman correlation. For estimating the p value, we used a nonparametric statistical permutation test because the nonoverlapping 1.5-sec epochs of HFA were interchangeable. Note that the magnitudes of saliency and HFA epochs were not significantly autocorrelated. The null distribution was made from 1000 iterations of surrogated trial labels. In each iteration, the maximum correlation between HFA and saliency magnitude was taken across all electrodes in an ROI (namely, the lateral PFC, the OFC, and the hippocampus) for each participant. The proportion of HFA–saliency magnitude correlation coefficients in the null distribution that was more than the observed correlation coefficient yielded the nonparametric corrected p value for the observed correlation. A p value of <.05 was considered significant.

Effects of Anticipating Salient Events

We tested if HFA–saliency magnitude correlation changed with anticipating salient events using a linear mixed-effects model. We recalculated the correlations between HFA and saliency magnitudes in each brain region in sliding windows of 15 sec with overlaps of 7.5 sec (24 bins). We used a linear mixed-effects model to test the effect of novel (n = 13) or anticipated periods of salient events (n = 11; rep = 0 for novel and 1 for repetitive storylines; Figure 4) from the three ROIs (the correlation coefficients [R] in each electrode region; the hippocampus, OFC, and dorsolateral PFC). We used the linear mixed-effects model in MATLAB (fitlme) to account for the different number of electrodes in each ROI of a subject and the nested effect of time: formulated as R ∼ rep + (rep | subject : electrodes) + (1 | rep : time_bin). ANOVA was applied for the result of F tests for the fixed-effect term in the linear mixed-effects model.

Electrode Localization and Visualization

Electrode locations were reconstructed and visualized in MATLAB using the Fieldtrip toolbox (Stolk et al., 2018). We manually selected electrodes on the postimplantation CT, which was coregistered to the preimplantation MRI using SPM (Ashburner & Friston, 1997) to maximize the accuracy of the reconstructions. A neurologist identified the electrodes' locations. We then normalized each patient's preimplantation MRI to the MNI-152 template brain using SPM to obtain the electrode positions in MNI space (Ashburner & Friston, 1999). If electrode locations in MNI space did not correspond to electrode locations in native (participant) space after normalization (e.g., an electrode is within hippocampus in native space but appears outside the hippocampus in MNI space after normalization), then electrode locations were manually adjusted to represent their true locations in native space. Electrode locations for bipolar re-referenced channels were calculated as the midpoint between the two electrodes (Burke et al., 2013, 2014; Long, Burke, & Kahana, 2014). Representations of the cerebral cortex were generated using FreeSurfer (Dale, Fischl, & Sereno, 1999) and representations of the hippocampus were generated from the Desikan–Killiany atlas (Desikan et al., 2006) using Fieldtrip. Brodmann's areas were inferred from Bioimage Suite package (bioimagesuite.yale.edu/).

RESULTS

We identified the magnitude of salient events in the movie by studying a separate group of adults (n = 80). This group indicated when, during the movie, a new episode started (i.e., perceiving an event boundary). The metric of event saliency magnitude was the proportion of identified event boundaries in a short time window of the movie (about 20 frames or 1.5 sec; Figure 1B; total movie time was 3 min). Epochs of 1.5 sec resulted in saliency magnitude of 0–0.6 (1 would be the maximum saliency magnitude when every participant agrees that, during the same 1.5 sec, an event boundary occurred). A distance matrix was constructed from the sum of the saliency of events that occurred between pairs of events and was used for binary hierarchical event clustering (see the Methods section; Figure 1). We tested the hypothesis that the magnitude of event saliency was tracked in the targeted regions. Ranked (Spearman) correlation and nonparametric permutation tests for statistical results were applied. The interchangeable nonoverlapping HFA in 1.5-sec epochs allowed using nonparametric permutation testing. Cluster-corrected p values are reported.

We observed that HFA correlated with the magnitude of event saliency was captured by the behavior of the independent rating group. The neural effect was clustered in dorsolateral PFC (BA 6, BA 8, BA 9, and BA 10; in five of seven patients; seven of nine patients with lateral PFC electrodes had dorsolateral PFC coverage). The effect was also detected in the hippocampus (BA 54; in three of three patients) and the medial OFC (BA 11; in three of three patients with medial OFC coverage; three of seven patients with OFC electrodes had medial PFC coverage; Figure 3). See Figure 3 for the correlation coefficient of all electrodes and Table 2 for statistical results of electrodes that showed a significant correlation (the R value of electrodes with p > .05 is color-coded in Figure 3).

Figure 3. 

Event saliency was tracked in the dorsolateral PFC, the medial OFC, and the hippocampus. (A) HFA in the dorsolateral PFC and the medial OFC increased with increasing event saliency. The electrodes are color-coded by the Spearman correlation coefficient (R) between HFA and event saliency across 1.5-sec epochs, which ranged between −0.3 and 0.7. (B) The saliency of each epoch (gray line) and HFA across epochs in a left medial OFC electrode for example (black line). For demonstration, HFA and saliency were smoothed by a 10-episode window (equal to 15 sec). The underlying data were not autocorrelated. (C) The same as B but for a right lateral PFC electrode. (D) HFA in the hippocampus also correlated with event saliency magnitude. (E) An example of HFA in the hippocampus and the epochs saliency magnitude (same as B and C). (A and D) The black outlines highlight the electrodes that showed the effect (cluster-corrected p < .05). The thickness of the black outline reflects the effect significance (see Table 2 for the exact p values < .05); the arrows show which electrode corresponds to the Plots B, C, and E, which were from Patient 6.

Figure 3. 

Event saliency was tracked in the dorsolateral PFC, the medial OFC, and the hippocampus. (A) HFA in the dorsolateral PFC and the medial OFC increased with increasing event saliency. The electrodes are color-coded by the Spearman correlation coefficient (R) between HFA and event saliency across 1.5-sec epochs, which ranged between −0.3 and 0.7. (B) The saliency of each epoch (gray line) and HFA across epochs in a left medial OFC electrode for example (black line). For demonstration, HFA and saliency were smoothed by a 10-episode window (equal to 15 sec). The underlying data were not autocorrelated. (C) The same as B but for a right lateral PFC electrode. (D) HFA in the hippocampus also correlated with event saliency magnitude. (E) An example of HFA in the hippocampus and the epochs saliency magnitude (same as B and C). (A and D) The black outlines highlight the electrodes that showed the effect (cluster-corrected p < .05). The thickness of the black outline reflects the effect significance (see Table 2 for the exact p values < .05); the arrows show which electrode corresponds to the Plots B, C, and E, which were from Patient 6.

Table 2. 
Statistical Results
Patientrpxyz
Hippocampus 
 1 .257 .021 −32.42 −17.61 −13.84 
 1 .251 .025 −32.85 −33.69 −5.65 
 5 .273 .006 22.62 −15.30 −18.68 
 5 .217 .044 35.49 −16.88 −14.58 
 5 .230 .031 22.39 −29.53 −9.728 
 6 .246 .011 25.13 −21.11 −13.12 
 6 .283 .002 33.78 −20.60 −11.82 
  
OFC 
 1 .305 .003 16.26 45.28 −22.26 
 1 .271 .005 20.11 44.06 −19.80 
 4 .263 .004 19.01 47.55 −8.65 
 6 .410 < .001 −14.80 34.05 −27.56 
 6 .456 < .001 −17.17 36.55 −20.63 
 6 .241 .009 −18.54 39.42 −12.64 
  
Dorsolateral PFC 
 2 .280 .012 −45.65 −0.92 32.36 
 4 .350 .003 26.82 41.96 51.74 
 4 .391 .001 32.04 23.65 59.05 
 4 .352 .003 33.25 23.40 63.07 
 4 .349 .003 −32.37 25.08 60.27 
 6 .369 .001 30.23 40.62 −2.91 
 6 .279 .013 32.44 43.05 4.80 
 6 .602 < .001 37.97 1.86 30.73 
 6 .539 < .001 37.01 7.27 36.46 
 6 .409 < .001 36.53 13.01 41.85 
 6 .308 .005 36.32 18.54 47.24 
 6 .268 .022 −40.57 13.40 44.16 
 6 .302 .005 −26.79 29.02 26.59 
 6 .291 .005 −40.40 27.15 30.77 
 6 .397 < .001 −37.94 −12.53 42.31 
 6 .465 < .001 −45.67 −12.97 43.13 
 6 .336 .001 −52.64 −13.45 43.82 
 9 .231 .013 18.2 7.735 49.02 
 9 .247 .008 −37.13 14.50 52.60 
 9 .215 .025 −42.09 15.50 53.60 
 10 .245 .024 21.50 39.55 32.72 
 10 .258 .013 −35.98 13.74 25.19 
Patientrpxyz
Hippocampus 
 1 .257 .021 −32.42 −17.61 −13.84 
 1 .251 .025 −32.85 −33.69 −5.65 
 5 .273 .006 22.62 −15.30 −18.68 
 5 .217 .044 35.49 −16.88 −14.58 
 5 .230 .031 22.39 −29.53 −9.728 
 6 .246 .011 25.13 −21.11 −13.12 
 6 .283 .002 33.78 −20.60 −11.82 
  
OFC 
 1 .305 .003 16.26 45.28 −22.26 
 1 .271 .005 20.11 44.06 −19.80 
 4 .263 .004 19.01 47.55 −8.65 
 6 .410 < .001 −14.80 34.05 −27.56 
 6 .456 < .001 −17.17 36.55 −20.63 
 6 .241 .009 −18.54 39.42 −12.64 
  
Dorsolateral PFC 
 2 .280 .012 −45.65 −0.92 32.36 
 4 .350 .003 26.82 41.96 51.74 
 4 .391 .001 32.04 23.65 59.05 
 4 .352 .003 33.25 23.40 63.07 
 4 .349 .003 −32.37 25.08 60.27 
 6 .369 .001 30.23 40.62 −2.91 
 6 .279 .013 32.44 43.05 4.80 
 6 .602 < .001 37.97 1.86 30.73 
 6 .539 < .001 37.01 7.27 36.46 
 6 .409 < .001 36.53 13.01 41.85 
 6 .308 .005 36.32 18.54 47.24 
 6 .268 .022 −40.57 13.40 44.16 
 6 .302 .005 −26.79 29.02 26.59 
 6 .291 .005 −40.40 27.15 30.77 
 6 .397 < .001 −37.94 −12.53 42.31 
 6 .465 < .001 −45.67 −12.97 43.13 
 6 .336 .001 −52.64 −13.45 43.82 
 9 .231 .013 18.2 7.735 49.02 
 9 .247 .008 −37.13 14.50 52.60 
 9 .215 .025 −42.09 15.50 53.60 
 10 .245 .024 21.50 39.55 32.72 
 10 .258 .013 −35.98 13.74 25.19 

The table lists the patient's number, Spearman correlation coefficient r, cluster-corrected p value, and the MNI coordinates of the electrode in millimeters and RAS (positive x = right, positive y = anterior, and positive z = superior). The electrodes are sorted according to their localization into the hippocampus, OFC, or PFC, as observed in the MRI scan in native space.

We conducted a planned analysis on the electrodes that showed a significant correlation (i.e., task relevant) to assess the effects of anticipation (Table 2). We recalculated the correlation coefficient between HFA and event saliency magnitude in 15-sec-long sliding windows (7.5-sec overlaps), resulting in 24 tested windows, of which 11 had repetitive storylines and 13 were novel. The flow of salient events in 46% of the sliding windows was anticipated. A storyline was predictable if the same type of event reoccurred more than twice, such as repetition of animals going back and forth or repetition of scoring in a competition. The results of a linear mixed-effect model showed that the correlation coefficient between HFA and saliency magnitude was higher in the OFC when the salient events were anticipated than when they were novel (Figure 4A; OFC, F(1, 142) = 4.3, p = .039), and this effect was observed in all patients with task-relevant electrodes (Figure 4B). There was no difference between novel and anticipated salient events in the hippocampus, F(1, 166) = 0.39, p = .52, or the dorsolateral PFC, F(1, 526) = 1.26, p = .26.

Figure 4. 

Representation of event saliency during repetitive and novel storylines. HFA–saliency magnitude correlation coefficients (R) in sliding windows of 15 sec (with 7.5-sec overlaps) throughout the movie. (A) The solid lines show the mean R in the hippocampus (in magenta), medial OFC (in blue), and dorsolateral PFC (in cyan) in the y-axis. The x-axis is the time in minutes. The shaded lines show the SEM. The dashed line shows 1 for the periods with repetitive storylines (anticipated salient events) and 0 for novel periods. (B) The R across the time bins is color-coded in each electrode that was included in the planned test. The x-axis is the time in minutes. Each row shows R in an electrode. The patients' number, showing the owner of the electrode, is written on the y-axis. The R in the OFC was higher for anticipated salient events than for novel events (p < .05).

Figure 4. 

Representation of event saliency during repetitive and novel storylines. HFA–saliency magnitude correlation coefficients (R) in sliding windows of 15 sec (with 7.5-sec overlaps) throughout the movie. (A) The solid lines show the mean R in the hippocampus (in magenta), medial OFC (in blue), and dorsolateral PFC (in cyan) in the y-axis. The x-axis is the time in minutes. The shaded lines show the SEM. The dashed line shows 1 for the periods with repetitive storylines (anticipated salient events) and 0 for novel periods. (B) The R across the time bins is color-coded in each electrode that was included in the planned test. The x-axis is the time in minutes. Each row shows R in an electrode. The patients' number, showing the owner of the electrode, is written on the y-axis. The R in the OFC was higher for anticipated salient events than for novel events (p < .05).

DISCUSSION

During encoding, a sequence of events is segmented to construct a hierarchical representation of event associations (Kurby & Zacks, 2008; Zacks & Swallow, 2007; Zwaan & Radvansky, 1998), with clusters of associated events represented in lower levels of a hierarchy and the associations of the clusters of events represented in higher levels of the hierarchy. The construction of such a hierarchical association requires linking relevant events and separating events that occur in different circumstances. For instance, a circumstance changes with perceiving a deviant event. Detecting the magnitude of event saliency can also contribute to establishing the structure of associations. When the newly perceived event is not salient, the event is closely associated with the preceding events; however, if the new event is highly salient, it should be separated from the preceding events. Here, we report distributed neural regions that detect the magnitude of deviance (i.e., event saliency) in a flow of events, including dorsolateral PFC and hippocampus, and further show that anticipating the deviant events affects the OFC activities.

We used event segmentations of a large control population (behavior group) who watched the silent movie to infer the event segmentation in another group with intracranial electrodes (iEEG group) who watched the same movie. The behavior group's event segmentation provided the event saliency of the entire movie. The RT of each participant in this group was estimated from a target detection task and used for normalizing the timing of event boundaries (see the Methods section). We inferred the saliency from the proportion of people that reported an event boundary in each movie epoch. The 1.5-sec windows provided interchangeable epochs of data for using correlation and permutation tests (see the Methods section).

The iEEG group passively watched a movie and did not know about the segmentation task, allowing us to study spontaneous and naturalistic neural processing during parsing a continuous flow of events. We observed that the HFA that is linked to nearby single neural activity (Rich & Wallis, 2017; Lachaux et al., 2012; Jacobs & Kahana, 2009; Belitski et al., 2008; Ray, Crone, et al., 2008) increased proportionally with event saliency in the hippocampus, dorsolateral PFC, and medial OFC.

HFA in the dorsolateral PFC in the iEEG group tracked event saliency magnitude. Dorsolateral PFC is critical for guiding attention (Corbetta & Shulman, 2002; Hopfinger, Buonocore, & Mangun, 2000; Kastner, Pinsk, De Weerd, Desimone, & Ungerleider, 1999; Paus, 1996), and increased HFA may, in part, be due to attention to novel events (Ray, Niebur, Hsiao, Sinai, & Crone, 2008; Zacks et al., 2001). Dorsolateral PFC is also engaged in cognitive control and conflict monitoring (Miller & Cohen, 2001; MacDonald, Cohen, Stenger, & Carter, 2000) by detecting new associations of categories and exemplars (Dolan & Fletcher, 1997). Accordingly, the observed additional correlation between HFA and event saliency in dorsolateral PFC reflects the demand for event segmentation and updating the event circumstance (Reynolds, Zacks, & Braver, 2007; Zacks et al., 2007).

Event saliency also correlated with HFA in the hippocampus. Hippocampal activity has been linked to the representation of event associations (Mack, Love, & Preston, 2017; Quiroga, 2012; Quiroga, Reddy, Kreiman, Koch, & Fried, 2005; Ekstrom et al., 2003). The hippocampal representation changes with salient changes in the environment (Shapiro, Tanila, & Eichenbaum, 1997), and its activity increases with detecting salient events (Chen, Cook, & Wagner, 2015; Chen et al., 2013; Axmacher et al., 2010; Kumaran & Maguire, 2007; Wittmann, Bunzeck, Dolan, & Düzel, 2007; Lisman & Otmakhova, 2001; Knight, 1996). Recent studies showed that hippocampal representations reflect the scale of topological saliencies of an environment, such as changes in the spatial closeness of streets or the centrality of the streets (Javadi et al., 2017), and the scale of deviance from expectation (Chen et al., 2015). Also, the hippocampal BOLD signal tracked the saliency of event boundaries when people watched movies (Ben-Yakov & Henson, 2018). Here, we propose that small deviance induces only a minor change in the hippocampal representation so that close events share more similar hippocampus representations than far events (Ezzyat & Davachi, 2014). These results expand the pattern separation mechanism for distinguishing similar visual associations attributed to the hippocampus (Yassa & Stark, 2011) to a mechanism for identifying the scale of event separation. Pattern separation for visual stimuli engages dentate gyrus in the hippocampus (Baker et al., 2016; Berron et al., 2016), but what the subregion of the human hippocampus contributes to the deviant detection is unknown (see Lisman & Grace, 2005, for the novelty signal in the rodent's subiculum and Knierim & Neunuebel, 2016, for mismatch signal in subregions of rodent's hippocampus).

We also observed a similar saliency magnitude effect in the medial OFC (BA 11 but not in the lateral OFC), with increased HFA for highly salient events. This observation is akin to the representation of saliency in the nonhuman primates' OFC, captured by HFA (Rich & Wallis, 2016, 2017). In humans, breaching expectations increases the OFC activity (Mikutta et al., 2015; Duarte, Henson, Knight, Emery, & Graham, 2009; Nobre et al., 1999). OFC also represents the saliency of anticipated events (Metereau & Dreher, 2015; Bechara, Tranel, Damasio, & Damasio, 1996). The reflection of the saliency magnitude suggests that OFC represents the structure of the event association. Notably, anticipation is a critical feature for encoding sequences of events because the gist of a previous experience shapes the expected context (Reynolds et al., 2007; Purcell, 1986). Here, the HFA in medial OFC tracked the event saliency better when the salient events were anticipated than during a novel intrusion of events, suggesting that the anticipated structure of event associations is represented in the medial OFC.

An important question concerns the dynamics of interaction in the neural network for representing the structure of event associations. For example, disturbing the input from the hippocampus to OFC impairs representing task structures in rodents (Wikenheiser, Marrero-Garcia, & Schoenbaum, 2017). It is not clear whether hippocampal deviancy detection is essential for constructing the OFC signal in humans or whether other brain regions such as midbrain structures contribute to detecting the magnitude of deviance (Dürschmid et al., 2016; Wittmann et al., 2007). For instance, hippocampal response to deviant events is associated with activity in the substantia nigra and ventral tegmental area (Murty & Adcock, 2014; Wittmann et al., 2007). It is also suggested that the hippocampus provides a novelty signal to the nucleus accumbens (Dürschmid et al., 2016). Although hippocampus activity is linked to anticipation (Jafarpour et al., 2017; Hindy et al., 2016; Hsieh et al., 2014), an outstanding question is whether the prediction is made by the hippocampus or is under control of other brain regions, such as the PFC. Systematically, comparison of regional activity, however, requires simultaneous recordings from both regions in a patient. A caveat of this iEEG study is that all the patients did not have sufficient coverage from multiple task-relevant regions to definitively study the dynamics of the network.

Event segmentation requires tracing the association of a new event to the preceding events (Zwaan & Radvansky, 1998). Detecting a new event's deviance magnitude helps with an accurate association of the event to the preceding sequence. Accordingly, segmented sequences that are separated by small surprises are more associated in comparison to a sequence separated by a big surprise. Detecting deviant events is known to increase neural activity in the PFC and the hippocampus (Long, Lee, & Kuhl, 2016; Axmacher et al., 2010; Bunzeck, Dayan, Dolan, & Duzel, 2010; Kumaran & Maguire, 2007; Strange, Duggins, Penny, Dolan, & Friston, 2005; Knight, 1996). Both brain regions, albeit differently, represent the associations in an experimental setup (Wikenheiser & Schoenbaum, 2016; Wilson, Takahashi, Schoenbaum, & Niv, 2014; O'Keefe & Nadel, 1978; Tolman, 1948). Here, we showed that the hippocampus and PFC regions tracked the scale of event saliency in a movie, and in the medial OFC, this effect is stronger when the salient events were anticipated than for novel events. We propose that a core function of the hippocampus, dorsolateral PFC, and medial OFC network is to construct the event association structure, akin to a task structure (Wikenheiser & Schoenbaum, 2016; Wilson et al., 2014).

Acknowledgments

This research used statistical consulting resources provided by the Center for Statistics and the Social Sciences, University of Washington. This work was sponsored by the James S. McDonnell Foundation, National Institute of Neurological Disorders grant R37 NS21135 (to R. T. K.), and the University of California, Irvine School of Medicine Bridge Fund (to J. J. L.). The authors are indebted to the patients for their participation. We thank Jie Zheng and members of the Knightlab for helping with data collection. We also thank Prof. Elizabeth Buffalo for helpful discussions.

Reprint requests should be sent to Anna Jafarpour, Department of Physiology and Biophysics, University of Washington, 1705 NE Pacific St., HSB Box 357290, Seattle, WA 98195, or via e-mail: annaja@uw.edu.

REFERENCES

REFERENCES
Ashburner
,
J.
, &
Friston
,
K. J.
(
1997
).
The role of registration and spatial normalization in detecting activations in functional imaging
.
Clinical MRI/Developments in MR
,
7
,
26
28
.
Ashburner
,
J.
, &
Friston
,
K. J.
(
1999
).
Nonlinear spatial normalization using basis functions
.
Human Brain Mapping
,
7
,
254
266
.
Axmacher
,
N.
,
Cohen
,
M. X.
,
Fell
,
J.
,
Haupt
,
S.
,
Dümpelmann
,
M.
,
Elger
,
C. E.
, et al
(
2010
).
Intracranial EEG correlates of expectancy and memory formation in the human hippocampus and nucleus accumbens
.
Neuron
,
65
,
541
549
.
Baker
,
S.
,
Vieweg
,
P.
,
Gao
,
F.
,
Gilboa
,
A.
,
Wolbers
,
T.
,
Black
,
S. E.
, et al
(
2016
).
The human dentate gyrus plays a necessary role in discriminating new memories
.
Current Biology
,
26
,
2629
2634
.
Bechara
,
A.
,
Tranel
,
D.
,
Damasio
,
H.
, &
Damasio
,
A. R.
(
1996
).
Failure to respond autonomically to anticipated future outcomes following damage to prefrontal cortex
.
Cerebral Cortex
,
6
,
215
225
.
Belitski
,
A.
,
Gretton
,
A.
,
Magri
,
C.
,
Murayama
,
Y.
,
Montemurro
,
M. A.
,
Logothetis
,
N. K.
, et al
(
2008
).
Low-frequency local field potentials and spikes in primary visual cortex convey independent visual information
.
Journal of Neuroscience
,
28
,
5696
5709
.
Ben-Yakov
,
A.
, &
Henson
,
R.
(
2018
).
The hippocampal film-editor: Sensitivity and specificity to event boundaries in continuous experience
.
Journal of Neuroscience
,
38
,
10057
10068
.
Berron
,
D.
,
Schütze
,
H.
,
Maass
,
A.
,
Cardenas-Blanco
,
A.
,
Kuijf
,
H. J.
,
Kumaran
,
D.
, et al
(
2016
).
Strong evidence for pattern separation in human dentate gyrus
.
Journal of Neuroscience
,
36
,
7569
7579
.
Brown
,
T. I.
,
Carr
,
V. A.
,
LaRocque
,
K. F.
,
Favila
,
S. E.
,
Gordon
,
A. M.
,
Bowles
,
B.
, et al
(
2016
).
Prospective representation of navigational goals in the human hippocampus
.
Science
,
352
,
1323
1326
.
Bunzeck
,
N.
,
Dayan
,
P.
,
Dolan
,
R. J.
, &
Duzel
,
E.
(
2010
).
A common mechanism for adaptive scaling of reward and novelty
.
Human Brain Mapping
,
31
,
1380
1394
.
Burke
,
J. F.
,
Long
,
N. M.
,
Zaghloul
,
K. A.
,
Sharan
,
A. D.
,
Sperling
,
M. R.
, &
Kahana
,
M. J.
(
2014
).
Human intracranial high-frequency activity maps episodic memory formation in space and time
.
Neuroimage
,
85
,
834
843
.
Burke
,
J. F.
,
Zaghloul
,
K. A.
,
Jacobs
,
J.
,
Williams
,
R. B.
,
Sperling
,
M. R.
,
Sharan
,
A. D.
, et al
(
2013
).
Synchronous and asynchronous theta and gamma activity during episodic memory formation
.
Journal of Neuroscience
,
33
,
292
304
.
Chen
,
J.
,
Cook
,
P. A.
, &
Wagner
,
A. D.
(
2015
).
Prediction strength modulates responses in human area CA1 to sequence violations
.
Journal of Neurophysiology
,
114
,
1227
1238
.
Chen
,
J.
,
Dastjerdi
,
M.
,
Foster
,
B. L.
,
LaRocque
,
K. F.
,
Rauschecker
,
A. M.
,
Parvizi
,
J.
, et al
(
2013
).
Human hippocampal increases in low-frequency power during associative prediction violations
.
Neuropsychologia
,
51
,
2344
2351
.
Corbetta
,
M.
, &
Shulman
,
G. L.
(
2002
).
Control of goal-directed and stimulus-driven attention in the brain
.
Nature Reviews Neuroscience
,
3
,
201
215
.
Coulson
,
S.
,
King
,
J. W.
, &
Kutas
,
M.
(
1998
).
Expect the unexpected: Event-related brain response to morphosyntactic violations
.
Language and Cognitive Processes
,
13
,
21
58
.
Dale
,
A. M.
,
Fischl
,
B.
, &
Sereno
,
M. I.
(
1999
).
Cortical surface-based analysis, I. Segmentation and surface reconstruction
.
Neuroimage
,
9
,
179
194
.
Desikan
,
R. S.
,
Ségonne
,
F.
,
Fischl
,
B.
,
Quinn
,
B. T.
,
Dickerson
,
B. C.
,
Blacker
,
D.
, et al
(
2006
).
An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest
.
Neuroimage
,
31
,
968
980
.
Dolan
,
R. J.
, &
Fletcher
,
P. C.
(
1997
).
Dissociating prefrontal and hippocampal function in episodic memory encoding
.
Nature
,
388
,
582
585
.
Duarte
,
A.
,
Henson
,
R. N.
,
Knight
,
R. T.
,
Emery
,
T.
, &
Graham
,
K. S.
(
2009
).
Orbito-frontal cortex is necessary for temporal context memory
.
Journal of Cognitive Neuroscience
,
22
,
1819
1831
.
Dürschmid
,
S.
,
Zaehle
,
T.
,
Hinrichs
,
H.
,
Heinze
,
H. J.
,
Voges
,
J.
,
Garrido
,
M. I.
, et al
(
2016
).
Sensory deviancy detection measured directly within the human nucleus accumbens
.
Cerebral Cortex
,
26
,
1168
1175
.
Ekstrom
,
A. D.
,
Kahana
,
M. J.
,
Caplan
,
J. B.
,
Fields
,
T. A.
,
Isham
,
E. A.
,
Newman
,
E. L.
, et al
(
2003
).
Cellular networks underlying human spatial navigation
.
Nature
,
425
,
184
188
.
Ezzyat
,
Y.
, &
Davachi
,
L.
(
2014
).
Similarity breeds proximity: Pattern similarity within and across contexts is related to later mnemonic judgments of temporal proximity
.
Neuron
,
81
,
1179
1189
.
Hanson
,
C.
, &
Hirst
,
W.
(
1989
).
On the representation of events: A study of orientation, recall, and recognition
.
Journal of Experimental Psychology: General
,
118
,
136
147
.
Hindy
,
N. C.
,
Ng
,
F. Y.
, &
Turk-Browne
,
N. B.
(
2016
).
Linking pattern completion in the hippocampus to predictive coding in visual cortex
.
Nature Neuroscience
,
19
,
665
667
.
Hopfinger
,
J. B.
,
Buonocore
,
M. H.
, &
Mangun
,
G. R.
(
2000
).
The neural mechanisms of top–down attentional control
.
Nature Neuroscience
,
3
,
284
291
.
Hsieh
,
L. T.
,
Gruber
,
M. J.
,
Jenkins
,
L. J.
, &
Ranganath
,
C.
(
2014
).
Hippocampal activity patterns carry information about objects in temporal context
.
Neuron
,
81
,
1165
1178
.
Hsieh
,
L. T.
, &
Ranganath
,
C.
(
2015
).
Cortical and subcortical contributions to sequence retrieval: Schematic coding of temporal context in the neocortical recollection network
.
Neuroimage
,
121
,
78
90
.
Jacobs
,
J.
, &
Kahana
,
M. J.
(
2009
).
Neural representations of individual stimuli in humans revealed by gamma-band electrocorticographic activity
.
Journal of Neuroscience
,
29
,
10203
10214
.
Jafarpour
,
A.
,
Piai
,
V.
,
Lin
,
J. J.
, &
Knight
,
R. T.
(
2017
).
Human hippocampal pre-activation predicts behavior
.
Scientific Reports
,
7
,
5959
.
Javadi
,
A.-H.
,
Emo
,
B.
,
Howard
,
L. R.
,
Zisch
,
F. E.
,
Yu
,
Y.
,
Knight
,
R.
, et al
(
2017
).
Hippocampal and prefrontal processing of network topology to simulate the future
.
Nature Communications
,
8
,
14652
.
Kastner
,
S.
,
Pinsk
,
M. A.
,
De Weerd
,
P.
,
Desimone
,
R.
, &
Ungerleider
,
L. G.
(
1999
).
Increased activity in human visual cortex during directed attention in the absence of visual stimulation
.
Neuron
,
22
,
751
761
.
Knierim
,
J. J.
, &
Neunuebel
,
J. P.
(
2016
).
Tracking the flow of hippocampal computation: Pattern separation, pattern completion, and attractor dynamics
.
Neurobiology of Learning and Memory
,
129
,
38
49
.
Knight
,
R.
(
1996
).
Contribution of human hippocampal region to novelty detection
.
Nature
,
383
,
256
259
.
Kumaran
,
D.
, &
Maguire
,
E. A.
(
2007
).
Match mismatch processes underlie human hippocampal responses to associative novelty
.
Journal of Neuroscience
,
27
,
8517
8524
.
Kurby
,
C. A.
, &
Zacks
,
J. M.
(
2008
).
Segmentation in the perception and memory of events
.
Trends in Cognitive Sciences
,
12
,
72
79
.
Lachaux
,
J. P.
,
Axmacher
,
N.
,
Mormann
,
F.
,
Halgren
,
E.
, &
Crone
,
N. E.
(
2012
).
High-frequency neural activity and human cognition: Past, present and possible future of intracranial EEG research
.
Progress in Neurobiology
,
98
,
279
301
.
Lisman
,
J. E.
, &
Grace
,
A. A.
(
2005
).
The hippocampal-VTA loop: Controlling the entry of information into long-term memory
.
Neuron
,
46
,
703
713
.
Lisman
,
J. E.
, &
Otmakhova
,
N. A.
(
2001
).
Storage, recall, and novelty detection of sequences by the hippocampus: Elaborating on the SOCRATIC model to account for normal and aberrant effects of dopamine
.
Hippocampus
,
11
,
551
568
.
Long
,
N. M.
,
Burke
,
J. F.
, &
Kahana
,
M. J.
(
2014
).
Subsequent memory effect in intracranial and scalp EEG
.
Neuroimage
,
84
,
488
494
.
Long
,
N. M.
,
Lee
,
H.
, &
Kuhl
,
B. A.
(
2016
).
Hippocampal mismatch signals are modulated by the strength of neural predictions and their similarity to outcomes
.
Journal of Neuroscience
,
36
,
12677
12687
.
MacDonald
,
A. W.
, III
,
Cohen
,
J. D.
,
Stenger
,
V. A.
, &
Carter
,
C. S.
(
2000
).
Dissociating the role of the dorsolateral prefrontal and anterior cingulate cortex in cognitive control
.
Science
,
288
,
1835
1838
.
Mack
,
M. L.
,
Love
,
B. C.
, &
Preston
,
A. R.
(
2017
).
Building concepts one episode at a time: The hippocampus and concept formation
.
Neuroscience Letters
,
680
,
31
38
.
Metereau
,
E.
, &
Dreher
,
J. C.
(
2015
).
The medial orbitofrontal cortex encodes a general unsigned value signal during anticipation of both appetitive and aversive events
.
Cortex
,
63
,
42
54
.
Mikutta
,
C. A.
,
Dürschmid
,
S.
,
Bean
,
N.
,
Lehne
,
M.
,
Lubell
,
J.
,
Altorfer
,
A.
, et al
(
2015
).
Amygdala and orbitofrontal engagement in breach and resolution of expectancy: A case study
.
Psychomusicology: Music, Mind, and Brain
,
25
,
357
365
.
Miller
,
E. K.
, &
Cohen
,
J. D.
(
2001
).
An integrative theory of prefrontal cortex function
.
Annual Review of Neuroscience
,
24
,
167
202
.
Murty
,
V. P.
, &
Adcock
,
R. A.
(
2014
).
Enriched encoding: Reward motivation organizes cortical networks for hippocampal detection of unexpected events
.
Cerebral Cortex
,
24
,
2160
2168
.
Nobre
,
A. C.
,
Coull
,
J. T.
,
Frith
,
C. D.
, &
Mesulam
,
M. M.
(
1999
).
Orbitofrontal cortex is activated during breaches of expectation in tasks of visual attention
.
Nature Neuroscience
,
2
,
11
12
.
O'Keefe
,
J.
, &
Nadel
,
L.
(
1978
).
The hippocampus as a cognitive map
.
Oxford
:
Oxford University Press
.
Oostenveld
,
R.
,
Fries
,
P.
,
Maris
,
E.
, &
Schoffelen
,
J.-M.
(
2011
).
FieldTrip: Open source software for advanced analysis of MEG, EEG, and Invasive electrophysiological data
.
Computational Intelligence and Neuroscience
,
2011
,
156869
.
Paus
,
T.
(
1996
).
Location and function of the human frontal eye-field: A selective review
.
Neuropsychologia
,
34
,
475
483
.
Paz
,
R.
,
Gelbard-Sagiv
,
H.
,
Mukamel
,
R.
,
Harel
,
M.
,
Malach
,
R.
, &
Fried
,
I.
(
2010
).
A neural substrate in the human hippocampus for linking successive events
.
Proceedings of the National Academy of Sciences, U.S.A.
,
107
,
6046
6051
.
Purcell
,
A. T.
(
1986
).
Environmental perception and affect: A schema discrepancy model
.
Environment and Behavior
,
18
,
3
30
.
Quiroga
,
R. Q.
(
2012
).
Concept cells: The building blocks of declarative memory functions
.
Nature Reviews Neuroscience
,
13
,
587
597
.
Quiroga
,
R. Q.
,
Reddy
,
L.
,
Kreiman
,
G.
,
Koch
,
C.
, &
Fried
,
I.
(
2005
).
Invariant visual representation by single neurons in the human brain
.
Nature
,
435
,
1102
1107
.
Ray
,
S.
,
Crone
,
N. E.
,
Niebur
,
E.
,
Franaszczuk
,
P. J.
, &
Hsiao
,
S. S.
(
2008
).
Neural correlates of high-gamma oscillations (60–200 Hz) in Macaque local field potentials and their potential implications in electrocorticography
.
Journal of Neuroscience
,
28
,
11526
11536
.
Ray
,
S.
,
Niebur
,
E.
,
Hsiao
,
S. S.
,
Sinai
,
A.
, &
Crone
,
N. E.
(
2008
).
High-frequency gamma activity (80–150Hz) is increased in human cortex during selective attention
.
Clinical Neurophysiology
,
119
,
116
133
.
R Development Core Team
. (
2014
).
R: A language and environment for statistical computing
.
Vienna, Austria
:
R Foundation for Statistical Computing
. .
Reynolds
,
J. R.
,
Zacks
,
J. M.
, &
Braver
,
T. S.
(
2007
).
A computational model of event segmentation from perceptual prediction
.
Cognitive Science
,
31
,
613
643
.
Rich
,
E. L.
, &
Wallis
,
J. D.
(
2016
).
What stays the same in orbitofrontal cortex
.
Nature Neuroscience
,
19
,
768
770
.
Rich
,
E. L.
, &
Wallis
,
J. D.
(
2017
).
Spatiotemporal dynamics of information encoding revealed in orbitofrontal high-gamma
.
Nature Communications
,
8
,
1139
.
Rosani
,
A.
,
Boato
,
G.
, &
De Natale
,
F. G. B.
(
2015
).
EventMask: A game-based framework for event-saliency identification in images
.
IEEE Transactions on Multimedia
,
17
,
1359
1371
.
Schütz-Bosbach
,
S.
, &
Prinz
,
W.
(
2007
).
Prospective coding in event representation
.
Cognitive Processing
,
8
,
93
102
.
Shapiro
,
M. L.
,
Tanila
,
H.
, &
Eichenbaum
,
H.
(
1997
).
Cues that hippocampal place cells encode: Dynamic and hierarchical representation of local and distal stimuli
.
Hippocampus
,
7
,
624
642
.
Speer
,
N. K.
,
Zacks
,
J. M.
, &
Reynolds
,
J. R.
(
2007
).
Human brain activity time-locked to narrative event boundaries
.
Psychological Science
,
18
,
449
455
.
Stolk
,
A.
,
Griffin
,
S.
,
van der Meij
,
R.
,
Dewar
,
C.
,
Saez
,
I.
,
Lin
,
J. J.
, et al
(
2018
).
Integrated analysis of anatomical and electrophysiological human intracranial data
.
Nature Protocols
,
13
,
1699
1723
.
Strange
,
B. A.
,
Duggins
,
A.
,
Penny
,
W.
,
Dolan
,
R. J.
, &
Friston
,
K. J.
(
2005
).
Information theory, novelty and hippocampal responses: Unpredicted or unpredictable?
Neural Networks
,
18
,
225
230
.
Tolman
,
E. C.
(
1948
).
Cognitive maps in rats and men
.
Psychological Review
,
55
,
189
208
.
Whitney
,
C.
,
Huber
,
W.
,
Klann
,
J.
,
Weis
,
S.
,
Krach
,
S.
, &
Kircher
,
T.
(
2009
).
Neural correlates of narrative shifts during auditory story comprehension
.
Neuroimage
,
47
,
360
366
.
Wikenheiser
,
A. M.
, &
Schoenbaum
,
G.
(
2016
).
Over the river, through the woods: Cognitive maps in the hippocampus and orbitofrontal cortex
.
Nature Reviews Neuroscience
,
17
,
513
523
.
Wikenheiser
,
A. M.
,
Marrero-Garcia
,
Y.
, &
Schoenbaum
,
G.
(
2017
).
Suppression of ventral hippocampal output impairs integrated orbitofrontal encoding of task structure
.
Neuron
,
95
,
1197.e3
1207.e3
.
Wilson
,
R. C.
,
Takahashi
,
Y. K.
,
Schoenbaum
,
G.
, &
Niv
,
Y.
(
2014
).
Orbitofrontal cortex as a cognitive map of task space
.
Neuron
,
81
,
267
279
.
Wittmann
,
B. C.
,
Bunzeck
,
N.
,
Dolan
,
R. J.
, &
Düzel
,
E.
(
2007
).
Anticipation of novelty recruits reward system and hippocampus while promoting recollection
.
Neuroimage
,
38
,
194
202
.
Yassa
,
M. A.
, &
Stark
,
C. E.
(
2011
).
Pattern separation in the hippocampus
.
Trends in Neurosciences
,
34
,
515
525
.
Yeung
,
M.
,
Yeo
,
B.-L.
, &
Liu
,
B.
(
1996
).
Extracting story units from long programs for video browsing and navigation
. In
Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems
, pp.
296
305
.
Zacks
,
J. M.
,
Braver
,
T. S.
,
Sheridan
,
M. A.
,
Donaldson
,
D. I.
,
Snyder
,
A. Z.
,
Ollinger
,
J. M.
, et al
(
2001
).
Human brain activity time-locked to perceptual event boundaries
.
Nature Neuroscience
,
4
,
651
655
.
Zacks
,
J. M.
,
Speer
,
N. K.
,
Swallow
,
K. M.
,
Braver
,
T. S.
, &
Reynolds
,
J. R.
(
2007
).
Event perception: A mind-brain perspective
.
Psychological Bulletin
,
133
,
273
293
.
Zacks
,
J. M.
, &
Swallow
,
K. M.
(
2007
).
Event segmentation
.
Current Directions in Psychological Science
,
16
,
80
84
.
Zarcone
,
A.
,
van Schijndel
,
M.
,
Vogels
,
J.
, &
Demberg
,
V.
(
2016
).
Salience and attention in surprisal-based accounts of language processing
.
Frontiers in Psychology
,
7
,
844
.
Zhang
,
D.
,
Han
,
J.
,
Jiang
,
L.
,
Ye
,
S.
, &
Chang
,
X.
(
2017
).
Revealing event saliency in unconstrained video collection
.
IEEE Transactions on Image Processing
,
26
,
1746
1758
.
Zwaan
,
R. A.
, &
Radvansky
,
G. A.
(
1998
).
Situation models in language comprehension and memory
.
Psychological Bulletin
,
123
,
162
185
.