Localizing Syntactic Composition with Left-Corner Recurrent Neural Network Grammars

Recent developments in computational linguistics and natural language processing have developed various kinds of computational models that can be employed to investigate neural computations in the human brain (e.g., Schrimpf et al., 2021), providing a new approach to the neurobiology of language (Hale et al., 2022). Specifically, computational models have played an important role to test linguistic theories against human brain activity, and the previous literature have examined whether natural languages are represented as hierarchical syntactic structures or linear word sequences (Chomsky, 1957; Everaert et al., 2015). For example, Frank et al. (2015) demonstrated that sequential models like recurrent neural networks (RNNs) successfully predict human electroencephalography (EEG) relative to context-free grammars (CFGs), suggesting that human language processing is insensitive to hierarchical syntactic structures. In contrast, the positive results of hierarchical models like CFGs and more expressive grammar formalisms like minimalist grammars and combinatory categorial grammars have also been confirmed against human EEG (Brennan & Hale, 2019) as well as functional magnetic resonance imaging (fMRI) (Brennan et al., 2016; Stanojević et al., 2023).

Moreover, the hybrid computational model of RNNs and CFGs has been proposed in the computational linguistics/natural language processing literature, namely recurrent neural network grammars (RNNGs; Dyer et al., 2016) which jointly generate word sequences and their syntactic structures via the syntactic composition. Interestingly, RNNGs outperformed sequential models like long short-term memory networks (LSTMs) in predicting not only syntactic dependencies (Kuncoro et al., 2018; Wilcox et al., 2019) and human eye movement (Wilcox et al., 2020; Yoshida et al., 2021), but also human brain activity like EEG (Hale et al., 2018) and fMRI (Brennan et al., 2020). These results indicate that RNNGs are the neurobiologically plausible computational model of human language processing.

However, the vanilla RNNG in Hale et al. (2018) and Brennan et al. (2020) has employed the top-down parsing strategy, which has been pointed out in the psycholinguistics literature as suboptimal especially for head-final/left-branching languages, and alternatively the left-corner parsing strategy has been proposed as the psychologically plausible parsing strategy (Abney & Johnson, 1991; Resnik, 1992). In addition, the recent result reported the positive results of the left-corner parsing strategy modeling self-paced reading and human eye movement (Oh et al., 2022).

In this article, building on this line of inquiry, we investigate not only whether hierarchical models like RNNGs better explain human brain activity than sequential models like LSTMs, but also which parsing strategy is more neurobiologically plausible. Specifically, there are two components in this paper. The first component is to construct a novel fMRI corpus named BCCWJ-fMRI where participants read newspaper articles selected from the Balanced Corpus of Contemporary Written Japanese (BCCWJ; Maekawa et al., 2014) through the naturalistic fMRI experiment. The second component is to evaluate computational models such as LSTMs, top-down RNNGs, and left-corner RNNGs against the novel fMRI corpus developed above. Importantly for the purpose here, given that Japanese is a head-final/left-branching language, this language should serve as an excellent testing ground to differentiate top-down and left-corner parsing strategies. To preview our results, we demonstrate that left-corner RNNGs outperform both LSTMs andtop-down RNNGs in the left inferior frontal and temporal-parietal regions, suggesting that there are certain brain regions that localize the syntactic composition with the left-corner parsing strategy.

In this subsection, we describe a novel fMRI corpus named BCCWJ-fMRI, that is, BCCWJ experimentally annotated with human fMRI.

Forty-two Japanese native speakers were recruited (19 females and 23 males, range: 18–24 years old, mean age = 21.1, SD = 1.7). At the time of the experiment, all of them were undergraduate and graduate students at Tohoku University, which is located in the northern part of Japan. All participants were right handed and had normal or corrected-to-normal vision without any neurological deficits. For each participant, written informed consent was obtained prior to the experiment.

Stimuli for this experiment consisted of 20 newspaper articles from the BCCWJ (Maekawa et al., 2014). BCCWJ consists of 100 million words, which includes various texts such as books, newspapers, blogs, laws, and so forth. Like BCCWJ-EEG (Oseki & Asahara, 2020), the newspaper articles were all segmented into phrasal units instructed by the National Institute for Japanese Language and Linguistics. The 20 newspaper articles were divided into four blocks (A, B, C, D). Each block lasted for around 7 min excluding the first 20 s that the stimuli were not presented and 31 s for reading and answering the comprehension questions.

During scanning, the stimuli were presented using rapid serial visual presentation (RSPVP) with PsychoPy (Peirce, 2007, 2009) where each segment was presented for 500 ms followed by a blank screen for 500 ms. Each participant read all blocks (A, B, C, D) in a randomized order. For each article, one yes–no comprehension question was given.

Scanning was conducted using the Philips Achieva 3.0T MRI scanner. During fMRI scanning, T2*-weighted MR signals were measured using a echo planar imaging pulse sequence (parameters: repetition time [TR] = 2,000 ms, echo time = 30 ms, flip angle = 80°, slice thickness = 4 mm, no slice gap, field of view = 192 mm, matrix = 64 × 64, and voxel size = 3 × 3 × 4). T1-weighted high-resolution anatomical images were also obtained (parameters: thickness = 1 mm, field of view = 256 mm, matrix = 368 × 368, repetition time = 1,100 ms, echo time = 5.1 ms) from each participant to use for preprocessing.

The obtained fMRI data were pre-processed using MATLAB (MathWorks, Natick, MA, USA) and Statistical Parametric Mapping (SPM12) software. The preprocessing included correction for head motion (realignment), slice timing correction, co-registration to theanatomical image, segmentation for normalization, spatial normalization using the Montreal Neurological Institute (MNI) template, and smoothing using a Gaussian filter with a full-width at a half-maximum (FWHM) of 8 mm.

5-gram models are a sequential model, which processes a word sequence without explicitly modeling its hierarchical structures. 5-gram models treat the context as a fixed window (Markov model), so it works as a weak sequential baseline for hierarchical models. We used 5-gram models (a fifth-order Markov language model with Keneser-Ney Smoothing) implemented with KenLM (Heafield, 2011).

LSTMs (Hochreiter & Schmidhuber, 1997) are a sequential model, which processes a word sequence without explicitly modeling its hierarchical structure. LSTMs can maintain the whole context as a single vector representation, so they work as a strong sequential baseline for hierarchical models. We used 2-layer LSTMs with 256 hidden and input dimensions. The implementation by Gulordava et al. (2018) was employed.

Recurrent neural network grammars (RNNGs) are a hierarchical model, which jointly models a word sequence and its syntactic structure. RNNGs rely on a stack LSTM to keep the previously processed partial parse and compress them into a single vector representation. At each step of processing, one of the following actions is selected:

• GEN: Generate a terminal symbol.

• NT: Open a nonterminal symbol.

• REDUCE: Close a nonterminal symbol that was opened by NT.

During a REDUCE action, the composition function based on the bidirectional LSTMs is executed; in both directions, constituents of the closed nonterminal are encoded and the single phrasal representation is calculated from the output of the forward and reverse LSTMs.

Two types of RNNGs were tested in our experiment; top-down RNNGs and left-corner RNNGs, namely, RNNGs that process the sentence and its syntactic structure in a top-down or left-corner fashion, respectively. We used RNNGs that had 2-layer stack LSTMs with 256 hidden and input dimensions. The implementation by Noji and Oseki (2021) was employed.

For inference of RNNGs, word-synchronous beam search (Stern et al., 2017) was employed. Word-synchronous beam search retains a collection of the most likely syntactic structures that are predicted given an observed partial sentence and marginalizes their probabilities to approximate the next word probability given the context. Although RNNGs can be employed in different beam sizes, we used the top-down RNNG with beam size k = 1,000 and the left-corner RNNG with beam size k = 400 for this study, based on Yoshida et al. (2021).

We utilized the computational models trained by Yoshida et al. (2021). Yoshida et al. (2021) trained these language models (LMs) on the National Institute for Japanese Language and Linguistics Parsed Corpus of Modern Japanese (2016), which comprises 67,018 sentences annotated with syntactic structures. The sequential LMs, the 5-gram model and LSTM, were trained with terminals only (i.e., word sequences), while hierarchical LMs, top-down RNNGs and left-corner RNNGs, were trained with terminals and their syntactic structures. See Yoshida et al. (2021) for the details of hyperparameter settings.

To quantify the quality of the models, the perplexity for each model was calculated. The models were computed for the texts that consist of 20 Japanese newspaper articles from BCCWJ. The perplexity for each model is as follows: 5-gram models (195.58), LSTMs (166.52), the top-down RNNG with beam size 1,000 (177.84), and the left-corner RNNG with beam size 400 (166.92). The full list of the perplexity for each LM, including different beam size RNNGs is summarized in the Table 1.

When the surprisal increases, there should be longer reading times or greater neural activities. In this study, we utilized the blood oxygen level-dependent (BOLD) signal as the measure of cognitive effort from humans.

In addition to surprisal, distance for RNNGs was employed in this study. This metric quantifies “syntactic work” where the number of parser actions (e.g., GEN, NT, REDUCE) is counted (Hale et al., 2018). Since RNNGs jointly model a word sequence and its syntactic structure, the word-synchronous beam search algorithm (Stern et al., 2017) is adopted to resolve the imbalance of the probability of the strings and the probability of the trees that RNNGs generate. This algorithm resolves this imbalance by considering “enough” potential parser actions. Distance is calculated by counting the number of these actions in the beam for each segment. Because this metric considers the number of actions in the beam, it is a more direct way of exploring the measure of cognitive effort of the syntactic processing in the brain.

Intuitively speaking, this metric is similar to the node count metric (e.g., Brennan et al., 2012, 2016), but not identical. These two metrics are similar in that they consider syntactic structures. The difference is that node count is applied to syntactic structures that are already constructed (i.e., a perfect oracle; cf. Brennan, 2016; Hale, 2014), whereas distance is counting the process and considering alternative structures that are potentially correct structures at the end of the sentence. Since this metric can only be employed for RNNGs, distance becomes relevant when RNNGs with different parsing strategies are compared in this study.

Before the statistical analysis, data from four participants were excluded due to an incomplete acquisition issue during the scanning in the MRI scanner (the scan stopped earlier than the designed time due to the experimenter’s error). Data from two participants were excluded due to the excessive head movement and data from two participants were excluded due to poor performance of the comprehension questions. Thus, data from 34 participants were used for data analysis.

Eight regions of interest (ROIs) in the left hemisphere were selected for this study based on previous work on the cognitive neuroscience of language literature (Bemis & Pylkkänen, 2011, 2013; Friederici, 2017; Hagoort, 2016; Matchin & Hickok, 2020; Zaccarella & Friederici, 2015). The ROIs chosen are the pars operularis (IFGoperc), the pars triangularis (IFGtriang), the pars orbitalis (IFGorb), the inferior parietal lobule (IPL), the angular gyrus (AG), the superior temporal gyrus (STG), the superior temporal pole (sATL), and the middle temporal pole (mATL). These regions were defined by automated anatomical labeling (AAL) atlas (Tzourio-Mazoyer et al., 2002). These regions are also motivated by the recent computational neurolinguistics literature (Brennan et al., 2016, 2020; Li & Hale, 2019; Lopopolo et al., 2017; Lopopolo et al., 2021; Stanojević et al., 2021). In order to extract the BOLD signals for the ROI analyses, the parcellation was provided by AAL Atlas using nilearn (Version 0.9.2; Abraham et al., 2014; Nilearn, 2010; Pedregosa et al., 2012), a Python package for statistical analysis of neuroimaging data.

In this work, we used control predictors that are not our theoretical interests but yet reflect human language processing. Word rate (word_rate) is an indicator that assigns 1 to the offset of the segment that was presented in the screen for 500 ms and 0 elsewhere. This predictor tracks the rate at which the segment is presented during participants read segments, which covers the broad brain activities that have to do with language comprehension (cf. Brennan et al., 2012). Word length (word_length) was also used as a predictor for the baseline model, which counts the number of characters for each segment. Word frequency (word_freq) is a predictor for the log mean of the word frequencies for each segment. The value of sentence ID (sentid) is the number that was assigned to sentences in each block and the value of the sentence position (sentpos) indicates the number of the position of segments within a sentence for each article. Overall, we included 11 control predictors including six head movement parameters (dx, dy, dz, rx, ry, rz).

The predictors of our theoretical interests are the surprisal estimated from the 5-gram model and LSTM, the surprisal computed from the top-down RNNG (surp_RNNG_TD) and the left-corner RNNG (surp_RNNG_LC), and the distance computed from the top-down RNNG (dis_RNNG_TD) and the left-corner RNNG (dis_RNNG_LC). These predictors were transformed into estimated BOLD signals via a canonical hemodynamic response function (HRF) in order. (i) We created segment-by-segment time series for the values of surprisal computed from the 5-gram model, LSTM, and RNNGs, and time series for the values of distance estimated from RNNGs. (ii) These values as well as the values from control predictors (word_rate, word_length, word_freq, sentid, and sentpos) were convolved with the HRF using nilearn (more specifically, using the function compute_regressor). The head movement parameters were excluded from this computation. (iii) The convolved values from the 5-gram model, LSTM, and RNNGs were orthogonalized against word_rate to isolate each predictor’s effect from the broad language processing effects. (iv) compute_regressor was done with re-sampling the values to 0.5 Hz to match the time series of the fMRI data (TR = 2.0). After executing compute_regressor, the output was concatenated with the fMRI time series from 34 individuals in the eight ROIs that are extracted using AAL Atlas via nilearn.

In Table 2, the Pearson correlation matrix between predictors excluding six head movement parameters is shown.

Among predictors, word rate is highly correlated with word frequency (r(word rate, word freq) = 0.996) as well as word length (r(word rate, word freq) = 0.84). Word frequency and word length are also highly correlated (r(word freq, word length) = 0.83). Sentence ID is relatively correlated with word rate (r(word rate, sentid) = 0.68), word length (r(word length, sentid) = 0.67), and word frequency (r(word freq, sentid) = 0.69). The similar pattern can be seen for sentence position as well. In terms of predictors of our interests, 5-gram is highly correlated with LSTM and surp_RNNGs (r(5-gram, LSTM) = 0.98, r(5-gram, surp_RNNG_TD) = 0.98, and r(5-gram, surp_RNNG_LC) = 0.98). LSTM, and two surp_RNNGs are also highly correlated with each other (r(LSTM, surp_RNNG_TD) = 0.99, r(LSTM, surp_RNNG_LC) = 0.99, and r(surp_RNNG_TD, surp_RNNG_LC) = 0.99). The two predictors for distance are also relatively correlated (r(dis_RNNG_TD, dis_RNNG_LC) = 0.84), while these two predictors do not have a high correlation with the predictors such as 5-gram and LSTM (e.g., r(LSTM, dis_RNNG_LC) = 0.43).

Regarding distance, we constructed a regression model that includes the control predictors, 5-gram, and LSTM. Then we only added dis_RNNG_TD, and applied anova() to the model without dis_RNNG_TD and the model that includes dis_RNNG_TD. Then we added dis_RNNG_LC to the model to test whether the left-corner RNNG has above-and-beyond effects for the top-down RNNG. Model comparison was also done with a model that includes the control predictors, 5-gram, and LSTM, and a model that includes dis_RNNG_LC as well as the control predictors, 5-gram, and LSTM to test whether dis_RNNG_LC has above-and-beyond effect for LSTM. We also tested dis_RNNG_TD whether the top-down RNNG has above-and-beyond effects for the left-corner RNNG in the same way. The following list summarizes what this study tested in the ROI analyses. The boldface text indicates what we tested in this article.

1. baseline model < n-gram < LSTM < surp_RNNG_TD < surp_RNNG_LC

2. baseline model < n-gram < LSTM < surp_RNNG_LC < surp_RNNG_TD

3. baseline model < n-gram < LSTM < dis_RNNG_TD < dis_RNNG_LC

4. baseline model < n-gram < LSTM < dis_RNNG_LC < dis_RNNG_TD

In addition to the ROI analyses, we also did an exploratory analysis independently. This analysis confirms the regions that are activated with respect to each predictor. Using nilearn package, the design matrices were created for the first-level general linear model. All predictors were included except for head movement parameters. The participant coefficient map was saved for the second-level analysis.

For the second-level analysis, one-sample t tests were performed. The threshold maps were z-valued and the threshold was defined as follows; false discovery rate was α = 0.05 and a threshold of the cluster size was 100 voxels. For the masking, Yeo et al.’s (2011) cortical mask was used and a FWHM Gaussian smoothing (8 mm) was applied. AtlasReader (Notter et al., 2019) was used for identifying the regions of peaks for each cluster size.

The mean number of correct responses across participants for the comprehension questions was 13.6 (SD = 3.6) out of 20 (68%).

Table 3 shows the results of the model comparisons of 5-gram, LSTM, surp_RNNG_TD, and surp_RNNG_LC. These comparisons were done by sequentially adding terms of theoretical interests. We found no statistically significant effects across ROIs for both 5-gram and LSTM models. Furthermore, there are no statistically significant effects by just adding surp_RNNG_TD across ROIs. However, when surp_RNNG_LC was added and compared with the model without it, all ROIs except for mATL showed statistically significant effects even after corrected for multiple comparisons.

As Table 4 shows, we also tested whether surp_RNNG_LC has the above-and-beyond effects for LSTM. The results confirmed such effects in IFGoperc, IFGtriang, IPL, AG, and STG.

The next statistical analysis summarized in Table 5 shows that surp_RNNG_TD better fits to IFGoperc, IFGtriang, IPL, AG, STG, and sATL, compared to surp_RNNG_LC.

Regarding dis_RNNG_TD and dis_RNNG_LC, the results are summarized in Table 6. The results show that both dis_RNNG_TD and dis_RNNG_LC have statistically significant effects in several ROIs against LSTM; IFGoperc, IFGtriang, IPL, AG, and sATL for dis_RNNG_TD; and IFGoperc, IFGtriang, IFGorb, IPL, AG, STG, and sATL for dis_RNNG_LC respectively.

Table 7 shows the results for testing whether dis_RNNG_LC better explains the fMRI data than dis_RNNG_TD. The results showed statistically significant effects in IFGoperc, IFGtriang, IFGorb, IPL, AG, and STG. On the other hand, there were no statistically significant effects in any ROIs when we tested whether dis_RNNG_TD better fits to the fMRI data, compared to dis_RNNG_LC (Table 8).

Table 9 summarizes the results of ROI analyses in this study.

A reviewer raised the question whether the beam size differences for RNNGs make different results. In order to answer this question, we did model comparison analyses where a regression model that includes the control predictors as well as 5-gram and LSTM and a model that includes one RNNG as well as the control predictors, 5-gram, and LSTM were tested via anova() using (i) different beam sizes (k = 100, 200, 400, 600, 800, 1,000), (ii) different parsing strategies (top-down or left-corner), and (iii) different complexity metrics (surprisal and distance) of RNNGs. The details of the results are summarized in the Supporting Information, available at https://doi.org/10.1162/nol_a_00118. Overall, regardless of the beam size differences or complexity metrics, the left-corner RNNGs improve the model fit to the fMRI data, compared to LSTM. On the other hand, the surprisal estimated from top-down RNNGs only improve the model fit to the fMRI data when the beam size is small (k = 100, 200). The distance computed from top-down RNNGs improves the model fit to the fMRI data regardless of the beam size differences.

For the control predictors, the following results were obtained from the whole brain analysis (Table 10 and Figures 1–5).

The main results are reported as follows: Word rate (Figure 1) was associated with the activation in the bilateral fusiform gyri, bilateral middle occipital lobes, and the bilateral inferior frontal gyri (opercular part). Word length (Figure 2) was associated with the activation in the left lingual gyrus and the left middle temporal pole. Part of these results indicate that word rate and word length predictors are involved in the activities in the visual processing and the visual word form area.

Our main interests are the results of the whole brain analysis for LSTM, the top-down RNNG, and the left-corner RNNG, which are summarized in Table 11 (see also Figures 6–11).

The main results are as follows: As for LSTM, although the threshold is uncorrected, the increased activities were confirmed in the right middle temporal pole and the left IFGtriang (Figure 7). Notice that even though the AtlasReader indicates no_label, the increasing activity in the left posterior temporal lobe (PTL) can be observed in Figure 7. Surp_RNNG_TD was associated with the activities in the left fusiform gyrus and the right inferior occipital lobe (using an uncorrected threshold; see Figure 8). Surp_RNNG_LC was associated with activities in the right AG, the right middle temporal lobe and the left middle frontal gyrus (uncorrected; Figure 9). Dis_RNNG_TD was associated with activities in the left parietal lobule, the right AG as well as bilateral precuneus (Figure 10). As for dis_RNNG_LC (uncorrected; Figure 11), the main increased activities were observed in the left parietal lobule and the left IFGoperc.

Our goal for this study was to test not only whether RNNGs better explain human fMRI data than LSTMs, but also whether the left-corner RNNGs outperform the top-down RNNGs. We localized the syntactic composition effects of the left-corner RNNG in certain brain regions, using the information-theoretic metric, such as surprisal, and a metric that measures the syntactic work, that is, distance, to quantify the computational models. Surprisal is assumed to associate with the amount of the cognitive effort in the brain during language comprehension, which has been attested in the previous studies (Bhattasali & Resnik, 2021; Brennan et al., 2020; Henderson et al., 2016; Lopopolo et al., 2017; Willems et al., 2015). In Brennan et al. (2020), the surprisal estimated from LSTM had statistically significant effects for their ROIs such as the left ATL, the left IFG, the left PTL, and the left IPL, against a baseline model. However, our results did not show such effects for the 5-gram model and LSTM across all ROIs. We also adopted another complexity metric, distance, which was tested in Hale et al. (2018) and Brennan et al. (2020) for RNNGs. In Brennan et al. (2020), it was shown that distance calculated from the top-down RNNG had statistically significant effects in the left ATL, the left IFG, and the left PTL, compared to what they called RNNG-comp (a degraded version of RNNGs that does not include the composition function). In our results, dis_RNNG_LC showed statistically significant effects in the IFGoperc, IFGtriang, IFGorb, IPL, AG, STG and sATL, compared to LSTM (Table 6). Our results also found that dis_RNNG_TD improves the model fits to the fMRI data in the IFGoperc, IFGtriang, IPL, AG, and sATL, compared to LSTM. Considering these, we showed in addition to Brennan et al. (2020), that the hierarchical models better explain the fMRI data compared to sequential models.

The results of the whole brain analysis showed that some control predictors such as word rate and word length were involved in regions that are related to the visual processing and the visual word form area such as the fusiform gyrus and the occipital lobe. Since the task was reading sentences segment by segment, the activation of these regions is expected. In terms of sequential models, the activity in the left PTL was associated with LSTM. However, again, the ROI analyses did not show any statistically significant effects for 5-gram < LSTM, and it remains unclear how to interpret the activity in the left PTL for LSTM, at least in this study.

Although the surprisal estimated from the 5-gram model and LSTM did not fit the fMRI data well, the results of our ROI analyses showed that the left-corner RNNG had statistically significant effects in several ROIs, compared to LSTM (Table 4 and Table 6). These results suggest that the syntactic composition with the left corner parser strategy is involved in these regions, and our results align with the previous studies. For example, the surprisal computed from a top-down context-free parser in Henderson et al. (2016) was associated with the activities in the IFG including pars opercularis (BA44), compared to lexical surprisal. There is also a piece of evidence for STG associated with phrase structure grammar. Although they did not use surprisal, in Lopopolo et al. (2021), node count from structures generated by phrase structure grammar was used as a complexity metric, and it showed a significant effect in STG, whereas the dependency grammar (which describes the relationship between a head and its dependent) did not show such an effect in this region, but the middle temporal pole was responsible for this grammar. The result that the node count effect was shown in STG is compatible with our surp_RNNG_LC and dis_RNNG_LC results, but not compatible with the results of surp_RNNG_TD and dis_RNNG_TD. As mentioned above, on the other hand, Henderson et al. (2016) did show the effect in IFG for the surprisal computed from CFGs, but they also reported that they did not observe the effect in STG. These mixed results make it hard to evaluate the effect of STG, though it is considered to be involved in sentence-level comprehension (e.g., Nelson et al., 2017; Pallier et al., 2011).

The regions such as IFGoperc and IPL for dis_RNNG_LC appeared to be important based on our ROI analyses, and the whole brain analyses confirmed the strong activation in these regions. IFG has been attested in the literature in which a simple composition was examined (Friederici, 2017; Maran et al., 2022; Zaccarella & Friederici, 2015). However, several other studies suggest that there is no comprehensive understanding regarding the locus of the composition in the brain (Pylkkänen, 2019, 2020; Pylkkänen & Brennan, 2020). Our results from dis_RNNG_LC partially aligns with Brennan et al.’s (2020) results where the distance computed from top-down RNNGs had a significant effect in IFGoperc as well as in ATL and PTL in their results. Brennan and Pylkkänen (2017) showed that the left-corner CFG was associated with the activation in the left ATL, which our ROI analysis results did not show in the results of dis_RNNG_TD < dis_RNNG_LC (Table 7). However, the sATL effect for dis_RNNG_TD and dis_RNNG_LC was found against LSTM. This might indicate that sATL is involved in composition, but not involved in the effect of the left-corner parsing strategy, compared to the effect of the top-down parsing strategy.

So far, we have discussed the regions that were associated with the left-corner RNNG, but we have not discussed how surprisal or distance computed from the left-corner RNNG modulates in the brain. In previous studies, it has been unclear which brain region is responsible for which component of computational models since the role of the syntactic processing for each study has been observed using different grammars with different complexity metrics: for example, surprisal estimated from part-of-speech (Lopopolo et al., 2017); surprisal computed from CFGs (Henderson et al., 2016); node count from the structures generated by CFGs (Brennan et al., 2012; Brennan & Pylkkänen, 2017; Giglio et al., 2022; Lopopolo et al., 2021); node count from the structures generated by combinatory categorial grammars (Stanojević et al., 2021, 2023); node count from the structures generated by minimalist grammars (Brennan et al., 2016; Li & Hale, 2019); surprisal and distance computed from top-down RNNGs (Brennan et al., 2020). It might be a case where surprisal and the metrics that express the process of the steps (e.g., node count, distance) play roles in designated regions of the brain separately. For example, the steps of structure building might be involved in the PTL (Flick & Pylkkänen, 2020; Matar et al., 2021; Matchin & Hickok, 2020; Murphy et al., 2022), which is compatible with some previous studies (Brennan et al., 2016, 2020; Li & Hale, 2019; Stanojević et al., 2023). Surprisal, on the other hand, might be modulated in more broad regions that have to do with language processing in addition to the process of the steps. This point should be clarified in future work that can test different complexity metrics with different grammars or computational models using the same human data. Related to this discussion, the attempt for identifying the locus of composition has not been converged in the neurobiology of language literature; some studies have argued that a specific part of the Broca’s area is for syntactic composition (or merge; Zaccarella et al., 2017; Zaccarella & Friederici, 2015), while others have claimed that the ATL is the locus of semantic composition (Bemis & Pylkkänen, 2011, 2013; Zhang & Pylkkänen, 2015). Another candidate for the syntactic composition is the PTL (Flick & Pylkkänen, 2020; Matar et al., 2021; Matchin & Hickok, 2020; Murphy et al., 2022). Or, the connection between two regions (IFG and PTL) might be a source of syntactic composition (cf. Hardy et al., 2023; Maran et al., 2022; Wu et al., 2019). Although these candidates for syntactic composition are compatible with our results, future work needs to be done.

In this article, we investigated whether hierarchical models like RNNGs better explain human brain activity than sequential models like LSTMs, as well as which parsing strategy is more neurobiologically plausible. As a result, the surprisal metric computed from left-corner RNNGs significantly explained the brain regions including IFGoperc, IFGtriang, IPL, AG, and STG relative to LSTMs, though the surprisal metrics estimated from 5-gram models, LSTMs, and top-down RNNGs did not show any significant effects across eight regions in the ROI analyses. In addition, the distance metric computed from left-corner RNNGs did show significant effects in IFGoperc, IFGtriang, IFGorb, IPL, AG, and STG, relative to the distance metric estimated from top-down RNNGs, but notvice versa. Overall, our results suggest that left-corner RNNGs are the neurobiologically plausible computational model of human language processing, and there are certain brain regions that localize the syntactic composition with the left-corner parsing strategy.

We thank Haining Cui for fMRI data collection. We are also grateful to two anonymous reviewers for helpful suggestions and comments.

Yohei Oseki, Japan Society for the Promotion of Science (https://dx.doi.org/10.13039/501100000646), Award ID: JP21H05061. Yohei Oseki, Japan Society for the Promotion of Science (https://dx.doi.org/10.13039/501100000646), Award ID: JP19H05589. Yohei Oseki, Japan Science and Technology Agency (https://dx.doi.org/10.13039/501100002241), Award ID: JPMJPR21C2.

Yushi Sugimoto: Formal analysis: Lead; Investigation: Lead; Methodology: Equal; Software: Lead; Visualization: Lead; Writing – original draft: Lead. Ryo Yoshida: Conceptualization: Supporting; Writing – review & editing: Supporting. Hyeonjeong Jeong: Methodology: Supporting. Masatoshi Koizumi: Project administration: Lead. Jonathan R. Brennan: Methodology: Supporting. Yohei Oseki: Conceptualization: Lead; Funding acquisition: Lead; Methodology: Supporting; Project administration: Lead; Resources: Lead; Supervision: Lead; Writing – review & editing: Supporting.

The fMRI corpus will be made publicly available in the future. The statistical maps from the whole brain analyses are available on NeuroVault (https://identifiers.org/neurovault.collection:14567). The code for fMRI analyses is available at https://github.com/osekilab/RNNG-fMRI, which is modified from https://github.com/dgd45125/LPPxORCxEN-CN. The code for language models is available at https://github.com/osekilab/RNNG-EyeTrack.