Discontinuous Combinatory Constituency Parsing

Discontinuity is common in natural languages, as illustrated in Figure 1. Children from a discontinuous constituent are not necessarily consecutive because each can group with its syntactic cousins in the sentence rather than its two adjacent neighbors. Although this relaxation makes discontinuous parsing more challenging than continuous parsing, it becomes more valuable for studies and applications in non-configurational languages (Johnson, 1985), where word order does not determine grammatical function. With gradually saturated continuous parsing accuracy (Zhou and Zhao, 2019; Kitaev and Klein, 2018, 2020; Xin et al., 2021), discontinuous parsing has started gaining more attention (Fernández-González and Gómez-Rodríguez, 2020a, 2021; Corro, 2020).

Typically, constituency parsers are divided into two genres (not including methods employing dependency parsing, e.g., Fernández-González andGómez-Rodríguez [2020b]): 1) Global parsers use a fixed chart to search through all parsing possibilities for a global optimum. 2) Local parsers rely on fewer local decisions, which leads to lower complexities. The global parser complexities start at least from binary O(n³) (Kitaev and Klein, 2018) or m-ary O(n⁴) (Xin et al., 2021), resulting in low speeds for long parses. Global parsers introduce numerous hypotheses, whereas a local shift-reduce process or incremental parse exhibits more linguistic interests with fewer outputs (Yoshida et al., 2021; Kitaev et al., 2022). Neural global parsers dominate both continuous and discontinuous parsing (Corro, 2020; Ruprecht and Mörbitz, 2021) in terms of F1 score, but they do not exhibit a strong accuracy advantage over other parsers. Although local parsers may in some cases produce ill-formed trees, global parsers do not guarantee the selection of gold-standard answers from their charts.

We propose extending a pair of local parsers to achieve high speed, accuracy, and convenient investigation for both binary and multi-branching parses. Chen et al. (2021) proposed a pair of continuous parsers employing bottom-up vector compositionality. We dub these as neural combinatory constituency parsers (NCCP) with binary CB and multi-branching CM. To the best of our knowledge, they possess the top parsing speeds for continuous constituency. CB reflects a linguistic branching tendency, whereas CM represents unsupervised grammatical headedness through composition weight. CM is among the few parsers that do not require preprocessing binarization (Xin et al., 2021) but possess a high parsing speed. We dub our extension as DCCP with binary DB and multi-branching DM. The mechanisms with neural discontinuous combinators are shown in Figure 2.

Specifically, our combinators take a sentence-representing vector sequence as input and predict layers of concurrent tree-constructing actions. Two mechanisms are employed. DB triggers an action if the orientations of two neighboring vectors agree. A swap action exchanges the vectors; a joint action composes a new vector with them. CB only possesses a joint action. Meanwhile, DM takes discontinuous vectors to form biaffine attention matrices and decides their groups collectively; the remaining continuous vectors resort to chunking decisions, as with CM. NCCP and DCCP are unlexicalized supervised greedy parsers.

The contributions of our study are as follows:

• We propose a pair of discontinuous parsers¹ (i.e., binary and multi-branching) by extending continuous parsers of Chen et al. (2021).

• We demonstrate the effectiveness of our work on the Discontinuous Penn Treebank (Evang and Kallmeyer, 2011, DPTB) and the TIGER Treebank (Brants et al., 2004). Our parsers achieve new state-of-the-art discontinuous F1 scores and parsing speeds with a small set of training and inferring tricks, including discontinuity as data augmentation, unsupervised headedness, automatic hyperparameter tuning, and pre-trained language models.

The chart for binary continuous parsing (Kitaev and Klein, 2018) is triangular, as shown in black in Figure 3. The horizontal dimension enumerates the position of each node (i.e., each bit as a word). The vertical C indicates the number of continuous cases included in each node. Nodes at height h share combinatorics of complexity h. Consequently, a binary continuous chart parser has a fixed $Σh=0n(n−h)·h∼O(n3)$ complexity for the CKY decoding algorithm.

In m-ary and/or discontinuous cases (i.e., for multi-branching arity m and/or fan-out k of each constituent), the chart is superior to that of binary continuous parsing in terms of complexity. Both horizontal and vertical axes expand to diversify the combination of discrete bits of each lexical node. Because of the expansion, m-ary continuous global parsing (Xin et al., 2021) has O(n⁴) complexity, whereas binary discontinuous parsing has exponential complexity at O(n^3k) (Corro, 2020; Stanojevic and Steedman, 2020), where k ∈{1,2} are special cases for binary CFG (likely in Chomsky Normal Form [CNF]) and binary Linear Context-Free Rewriting Systems with maximum fan-out 2 (Stanojevic and Steedman, 2020, LCFRS-2). M-ary discontinuous parsing, which certainly has a higher complexity, is not yet available for global parsing.

For efficiency, expensive rules are commonly restricted by LCFRS-2 parsers (Corro, 2020; Stanojevic and Steedman, 2020; Ruprecht and Mörbitz, 2021). A tricky O(n³) variant of Corro (2020) covering major rules has produced the best results. However, the variant excludes 2% sophisticated discontinuous rules on TIGER Treebank. Limited by the simplified grammar, their discontinuous scores are low, especially for recalls. A global optimum does not guarantee a gold parse, leaving room for local parsing.

Local parsers do not observe the chart framework and only consider one greedy or a few hypotheses. Transition-based parsers with a swap or gap action have sequential actions and low complexities (Maier, 2015; Coavoux and Crabbé, 2017). Multiple swaps or gaps combine to construct a large discontinuous constituent. In contrast, stack-free parsing can directly pick up a distant component with one attachment search (Coavoux and Cohen, 2019). Easy-first (Nivre et al., 2009; Versley, 2014) and chunker-based (Ratnaparkhi, 1997; Collobert, 2011) run rapidly.

Fernández-González and Gómez-Rodríguez (2020a) redirected discontinuity to dependency parsing via pointer networks and obtained significant accuracy improvement among greedy parsers. However, it is difficult to determine whether such improvement originates from the model or the extra head information. In contrast, Fernández-González and Gómez-Rodríguez (2020b) reordered input words in at most O(n²) complexity and redirected to various continuous parsers.

We call a level of partial derivations or subtrees a ply (Jurafsky and Martin, 2009), as depicted in Figure 4. Similar to the state of a transition-based parser, each of which leads to an action, ply is the state for DCCP, each of which leads to a sequence of concurrent actions for itself.

Starting with a sequence of words (x₁,⋯ ,x_n) as an initial ply, we assemble an unlabeled discontinuous parse tree in a bottom–up manner by applying concurrent actions to the roots of subtrees in the ply and iterating until sequence length n = 1.

All nodes in a DB ply are derived by Formula 1 under the condition of Formula 2 to form a new ply. As exemplified in Figure 4 for DB, (x₁,x₂) and (x₄,x₅) meet the condition of Formula 2 and have respective joint and swap actions. Meanwhile, x₃ takes neither action and remains in the ply, because the orientations of x₂ and x₄ do not agree with x₃, regardless of x₃’s orientation.

Dozat and Manning (2017) characterized each dependency tree as a sparse asymmetric matrix via biaffine attention, with each sole positive signal in a row (or column) indicating a lexical dependency (from a word to its head or vice versa). Nevertheless, lexical dependency is not available for constituency parsing, and biaffine attention becomes expensive at O(n²) complexity.

Via fast chunking or a small biaffine attention matrix, DM balances to increase its efficiency. As exemplified in Figure 4 for DM, discontinuous (x₁,x₂,x₅) are grouped as one because of their mutual affinity, which is equivalent to a 3 × 3 biaffine attention matrix of ones. Node x₂ is selected as the medoid for the constituent’s location in the new ply. Meanwhile, continuous (x₃,x₄) forms a constituent for chunk(x_i ⊕ x_{i +1}) = affinity(x_i,x_{i +1}) = 𝟙(i∉{2,4}) and i ∈ [2,4].

Similar to a range of previous works (Chen et al., 2021; Shen et al., 2018; Kitaev and Klein, 2020; Corro, 2020), we adopt an empty label “$∅$” for our substructure (e.g., binarization). Additionally, we collapse each unary branch into a single node and join their constituent labels according to their hierarchical order (e.g., S+VP as derivation S$→$VP) with easy restoration during inference. Unary collapse is productive for label type, as shown in Table 1.

C children of a constituent join one by one via their orientation and joint signals. For c ∈ [1, C), [1, c]-th children are set to orientation right (1) and (c, C]-th are set to orientation left (0). Neighboring children have positive joint signals if they are siblings. Otherwise, negative joints swap them toward their siblings. We normalize a factor $ρDB=c−1C−2∈[0,1]$ for treebank binarization. In continuous parsing, ρ^DB ∈{0, 1} implies CNF.

As illustrated in Figure 5 (a) & (b), we obtain layers of action signals from the binarization of (c). As an extension to CNF, we use the beta distribution ρ^DB ∼Beta(α_left,α_right) ∈ (0, 1) to create augmented samples with $αleft,αright∈(0,+∞)$⁠.

We use a set of categorical medoid factors ρ^DM ∈{random,leftmost,rightmost} to stratify a multi-branching tree: 1) random picks a random child with uniform probability, whereas 2) leftmost and 3) rightmost take the two ends of a discontinuous group.

In Figure 5 (d), w₁ and w₄ are randomly selected as medoids. Meanwhile, $l21$ and $l22$ would exchange their places if w₃ and w₄ were selected. Medoid is different from headedness (Zwicky, 1985). It is an intermediate variable for locating a discontinuous constituent.

NCCP and DCCP have the same bottom–up iteration on a ply and share two types of neural components: bidirectional Long Short-Term Memory (BiLSTM) and feedforward neural network (FFNN).

In Algorithm 1, BiLSTM_cxt contextualizes the sequence of words x_1:n as embeddings $x1:n11$ and BiLSTM_ply contextualizes the ply sequence $x1:nhh$ for either DBFOLD in Algorithm 2 or DMFOLD in Algorithm 3, either of which modifies ply and constructs new layers of embeddings $x1:nh+1h+1$⁠. The necessity of BiLSTM_cxt and BiLSTM_ply contextualization was empirically examined with NCCP. Meanwhile, FFNN_tag and FFNN_label predict the lexical tags and constituent labels based on contextualized individual embeddings without grammar constraint.

The FOLD of DB is shown in Algorithm 2. The COMPOSE function uses sigmoid activation “σ” to create a pair of complementary gates λ and (1 − λ) for x_L and x_R. λ is a vector of the same size as the embeddings.

To identify discontinuous groups in the affinity biaffine attention matrix, DM takes $M=σâ[1:D,1:D]h$ for value range (0,1) (⁠$D=∑id^ih$⁠) and booleanizes it into B ← M > θ. It 1) tries default threshold θ = 0.5 as the natural selection for sigmoid activation and checks whether all the following statements are true:

• B is symmetric (i.e., $B=B⊺$⁠),

• any rows v,w ∈ B are v≠0,

• either v = w or $v⊺·w=0$⁠.

It succeeds in most cases. Otherwise, it 2) tries a value from M as θ, checks and loops again. We order the thresholds by their distances to the default 0.5. If all 2) fail, it 3) simply falls back into grouping all nodes as one and counts one FAIL.

Extreme cases provide the upper bounds for our theoretical complexity. DB takes $n2$ fully swapping plies and $n2$ fully joining plies. Each ply costs O(n) recurrency; the bound is O(n²). Every DM ply involves a matrix for all nodes and decreases n only by one. Assume that we limit check 2) to some fixed sizes. Each ply costs O(n²); the bound is O(n³).

However, DCCP has an empirical O(n²) complexity with strong linearity, as shown in Figure 6. DB has higher linear coefficients because of its slow binary combination. Meanwhile, DM shows stronger quadratic tendency because of biaffine attention. Yet, their coefficient magnitudes are on par with one another.

Figure 7 first summarizes (e) basic data augmentation with binarization for DB and medoid for DM (including (a), (b), and (d) in Figure 5). The beta distribution can resemble a uniform random distribution or other biased distributions to detect linguistic branching tendency with specific (α_left,α_right).

Then, we further leverage the intermediate non-terminal node “$∅$” to create more $∅$-subtrees. The augmentation is an inspiration of CM’s deterministic _SUB node, which balances subtree heights and boosts both accuracy and efficiency. (Non-_SUB trees remain at their original heights.) However, (g)$∅$-subtree is random and creates imbalance. It creates only one stretching branch by iteratively grouping nodes with possibility $ρ∅$⁠, which has three significant impacts:

• Random stretching branches add mild variations to the context as states for robust ply actions in FOLD.

• Random discontinuity creates DB orientation layers that cannot be created by ρ^DB binarization.

• They reduce large (possibly continuous) constituents into smaller (possibly discontinuous) pieces without adding a large payload to the biaffine attention, which narrows the gap between DB and DM (DM is more vulnerable to dramatic many-to-one COMPOSE).

Taking NP “a good day” for instance, any of “a day,” “a good,” and “good day” can be an intermediate option for creating the NP. On the one hand, these options create varied contexts for the remaining parts of a ply. On the other hand, assume that “a day” (which is not a ρ^DB product) is selected. DM learns to discern it with the other possible “a day” in biaffine attention based on their context.

To further randomize DB training, we introduce (f)ply shuffle and its resultant losses L_ori^shfl and L_jnt^shfl. It shuffles $x1:nhh$ with respect to each constituent, takes the new sequence to BiLSTM_ply and FOLD, and reuses the ply of orientation and joint for those additional losses. For example, a VP to the left of an NP gets shuffled to the right with the same ply “right$♦$left” producing the additional loss items.

Continuous affinity and discontinuous affinity in DM undergo different identification processes. To minimize the difference, we introduce (h)$LaffC$ and $LaffX$ for continuous and interply affinity, in addition to cardinal $LaffD$⁠. These reduce the risk of biaffine attention forwarding incorrect nodes, which would evoke exposure bias. We use positive rates β_c and β_x for $βc·σ(d^ih)$ and $βx·σ(xwh⊺·Waff·Δwh-+baff)$ to limit the sample size, where layers $h-≠h$ contain discontinuous nodes. Fallible signals are more likely to form losses via HINGE-LOSS.

DCCP takes frozen pre-trained FastText as static word embedding (PWE) or fine-tuned 12-layer pre-trained XLNet and BERT as contextualized embedding of pre-trained language models (PLM) as lexical input ² and parses on English DPTB and German TIGER treebanks. See Table 2.

PLM models also use general hyperparameters with learning rate 10⁻⁶ at S1. PLMs are frozen during the first 50 epochs to avoid noise pollution and then are fine-tuned with learning rate 3 × 10⁻⁶. They inherit explored hyperparameters from PWE models at S2, except for learning rate 3 × 10⁻⁶.

Table 3 shows F1 scores of recent neural discontinuous parsers under comparable conditions on test sets. We follow their reported number of significant digits and reduce the effects of random initialization with an average of five runs. The details are shown in Table 4.

DCCP models achieved state-of-the-art performance in terms of discontinuous F1 scores and parsing speeds. Although speed tests are conducted on different platforms, our parsers lead by a significant margin. In terms of overall F1 score, our parsers outperform some chart parsers (Stanojevic and Steedman, 2020; Ruprecht and Mörbitz, 2021) and slightly underperform the overall best outline, as characterized in boldface.

We ablate the PWE models in two-stage training, as shown in Table 5. We only show one representative run with ablation because of the similar low variability on development sets. DB has two data augmentation items $ρ∅$ and ρ^DB as well as one model item ply shuffle. On $ρ∅$ refers to $ρ∅=0.25$ and off $ρ∅=0$⁠. Off Beta(1,1) refers to a static ρ^DB = 0.5 and (0,0,0) shows the performance of bare DB models.

On the flip side, DM’s (0,0,0) contains randomness because of ρ^DM = random. We do not intend to examine a static ρ^DM as DB yields negative results. Based on effective training tricks, the variants enter the BO process at S2. DCCP shows its sensitivity to $ρ∅$ in Table 6.

Both CM and DM provide unsupervised headedness $λ-$⁠. Chen et al. (2021) were unable to test the benefits of CM’s unsupervised headedness because it is a final product that cannot affect parsing. However, DM’s medoid affects parsing performance. On PLM DM, we select different ρ^DM categories, which affect the location of all discontinuous constituent, and examine their generalization on test sets, as shown in Table 7. All models are trained with ρ^DM = random but inference with ρ^DM = uhead exerts positive gains on accuracy.

Table 3 exhibits high speeds and near state-of-the-art accuracies of DCCP compared to recent works. DCCP inherits many properties from NCCP. CB and CM are special cases of DB and DM without swap and discontinuous actions. All models contain compact components without grammar restriction. Each model has no more than 4.7M parameters apart from PWE or PLM, as listed in Table 2.

The variability of all models is low, except for the discontinuous F1 score of PLM DB on DPTB, as shown in Table 4. The main cause may not be the random initialization but the different training processes of PWE and PLM models—PLM models use the configuration of PWE models at S2. Those degraded PLM models adopted low $ρ∅$ configurations (e.g., $ρ∅=0.078$⁠). Because DPTB has less discontinuity and overall F1 scores are used for model evaluation, high variability in discontinuous F1 scores becomes more common without several BO trials; this phenomenon is also reflected in Table 5. DB lacks explicit discontinuity, and the selection of hyperparameters seems to be necessary on DPTB.

Figure 8 presents the F1 scores for discontinuity and multi-branching. We select PLM models whose overall F1 scores are close (i.e., most F1 differences are less than 0.1 and DB’s performance is high). DM exhibits persistent advantages over DB when these properties are frequent. We further determined that CM has the same gains over CB starting identically from 4-ary nodes with minor score differences on (D)PTB under the same $ρ∅=0$ condition, as shown in Table 9. The result supports the argument of Xin et al. (2021), which asserts that m-ary constituency parsing without binarization preserves some natural advantages, e.g., predicate-argument structure. Specifically, $ρ∅>0$ shifts DM’s multi-branching advantage to frequent low-arity trees, favoring the overall scores on DPTB, while it enhances both discontinuity and multi-branching advantages on TIGER, as shown in Table 10, in agreement with Table 6.

The training process of CB with CNF binarization has a slight impact on parsing accuracy. Chen et al. (2021) obtained the best CB with Bernoulli distribution P(ρ^DB = 0) = 0.85 (i.e., P(ρ^DB = 1) = 0.15 or L85R15 in their format “L%R%”) on PTB. They argued that such binarization brings orientation balance.

Similarly, DB’s S2 exhibits a slightly leftward exploration with the beta distribution, as shown in Figure 9. DPTB has a similar situation. Yet, the optimized distributions are relatively uniform and symmetric, which qualifies our uniform randomness with ρ^DB at S1 and indicates a desired property for future language-agnostic practice.

For unsupervised headedness, Chen et al. (2021) reported that CM absolutely picks determiners (DT) as heads for major NP. DM continues and further grammatically picks more noun phrases (NP) as NP heads, as shown in Table 8.

From Fig- ure 8, we learned that TIGER is more challenging in discontinuity. Incorrect discontinuity prediction seems to cascade to multi-branching prediction, resulting in degradation of both properties. Meanwhile, DPTB is largely transformed from PTB by typed traces and automatic rules, where the multi-branching accuracy stays more stable.

As seen in Table 7, ρ^DM = rightmost,leftmost yielded the second best F1 and D.F1 scores on DPTB and TIGER, respectively. The reversed setting yielded poor results, even if not the worst. The observation implies that many English heads locate rightward (right-branching) and that German heads tend to locate leftward (verb-second word order, V2). German has abundant separable verbs with their prefixes at the right-hand side of the clauses.

Greedy parsers allow ill-formed outputs without a single root, especially in case of single-model inference. Our models yielded a few invalid parses, as demonstrated in Table 11. DM models produce more errors. However, unsuccessful decomposition of biaffine attention matrices might not be the direct cause, as also shown in Figure 11. $ρ∅>0$ variants cleared the matrices that could not be decomposed with any θ (FAIL). Similar to CM, this genre suffers from more failures. Specifically, the ply size cannot be reduced to one during the iteration. Greedy parsers must suffer the defect because of their simplicity. However, invalid parses can contribute positive F1 scores and global parsers can yield inaccurate parses.

We applied methods such as Boolean matrix factorization and singular value decomposition. However, they did not provide any improvement and significantly slowed down the speed. This is because θ≠0.5 cases are few. Our sequential tries to decompose might be naïve, but they are effective. In Figure 11, the number of tries does not significantly increase under θ < 0.9 within 50 tries. For θ ≥ 0.9, although some tries are expensive, we will see that they are worthy in the next section. The imbalanced signals from both datasets account for the bias of θ—more than 92% of the biaffine attention signals are ones, as shown in Figure 12. All affinity biases (i.e., b^aff ∈ [−1.60,−0.84]) are significant negative numbers for counteraction.

The design of affinity as biaffine attention is an initial but coarse attempt, which brings imbalance. If one encoded dependency within constituent to biaffine attention instead, both signal balance and multi-grammar parsing might be better addressed. However, based on the context of constituency parsing, we leave this topic for future study.

Figure 13 highlights the value of discontinuous parsing by demonstrating the respective CM parse. Conspicuously, the branching tendency of the continuous parse is to the right, while it is not obvious for the discontinuous parsing. Meanwhile, we observed instances of similar unsupervised headedness weights. This sample is not trivial, which challenges our PLM DCCP models.

In Figure 13, DB shows sinuous travel traces of “I,” “was,” and larger $∅$-subtree nodes that involve the turning of orientations. The varying context leads them to achieve complex movement. DB also created some grammatical substructures for “How,” “referred,” “to,” “was,” “in,” and “school.”

Meanwhile, the DM parse is more dramatic. The formation of the lower discontinuous VP involves five nodes, two of which are irrelevant words “How” and “referred” triggered by incorrect discontinuity signals. They are discontinuous but for the higher VP. The two nodes create a noisy biaffine attention matrix because their grammatical roles are compatible with the lower VP. Trained for extra robustness, the matrix decomposition with five tries found the right θ to identify the correct VP members excluding “How” and “referred.” The interply loss and the decoding process gave this parse a chance for perfection.

In Figure 14 for German, DB achieved a long distant constituent in a more subtle way. The word “zwar” joins “registriert” as an $∅$-subtree when the formation of intermediate NP shortens their distance and prevents a travel through. “Gegenwärtig” follows and forms a VP. However, DM failed.

The above failure explains why DM is inferior to DB with $ρ∅=0$⁠. DB’s orientation system allows some free travel before joining with correct mates. The constituent formation through steps of accumulation creates a more stable context. However, DM’s group action happens all at once. An incorrect composition might create a quite different context, which leads to unseen reaction chains. The strange unsupervised headedness weights reflect the issue. On the other side, with $ρ∅>0$⁠, DM can also gradually build and discover some semantic substructures, as shown in Figure 15. In contrast, DB is not sensitive to $ρ∅$ because of its nature, in agreement with Table 6.

We proposed a pair of efficient and effective discontinuous combinatory constituency parsers and extended the neural combinator family of NCCP. A binary combinator DB is based on orientation extended into a joint-swap system. A multi-branching combinator DM leverages biaffine attention adapted for constituency. Our models (as in Table 2) achieved state-of-the-art discontinuous F1 scores with a significant advantage in speed.

In the future, we will aim to extend DCCP into a multilingual tool for directed acyclic graph (DAG) parsing with function tag prediction for predicate-argument structure.

We extend special thanks to our action editor, anonymous reviewers, and Prof. Yusuke Miyao for their invaluable comments and suggestions. This work was partly supported by TMU research fund for young scientists.