Assessing Corpus Evidence for Formal and Psycholinguistic Constraints on Nonprojectivity

The syntactic structure of natural language sentences can be captured to a large extent using dependency trees: directed trees drawn over words indicating what words are dependent on what other words (Tesnière 1959; Hays 1964; Mel’čuk 1988; Hudson 1990; Nivre 2015). An example is shown in Figure 1. A number of key formal questions in linguistics boil down to questions about the structure of these dependency trees. In particular, recent work has concluded that the characterization of natural language in formal language theory depends on the constraints that can be placed on crossing dependencies in dependency trees (Kuhlmann 2007).

Here we use recently available massively crosslinguistic dependency treebanks (Nivre et al. 2015; Gerdes et al. 2018, 2019) to take up the question, what distinguishes natural language dependency trees within the space of all possible tree structures, in terms of crossing dependencies? We investigate two kinds of proposed constraints on dependency trees. First, we investigate a set of graph-theoretic constraints originating in the formal language theory and dependency parsing literatures, which have bearing on the location of natural language within the Chomsky hierarchy. Comparing natural language dependency trees to random trees of various kinds, we find little evidence that linguistic trees are constrained by the kinds of formal properties studied in formal language theory literature, but we do find strong evidence for under-studied constraints originating from the parsing literature. Second, we explore a set of performance-based and psycholinguistically motivated soft constraints, motivated in terms of empirically measured human online processing difficulty, finding evidence that these constrain crossing dependencies.

The attempt to characterize the complexity of natural language in terms of formal language theory has been an extraordinarily productive enterprise joining linguistics, computer science, and mathematics (Chomsky 1956; Chomsky and Schützenberger 1963; Hopcroft and Ullman 1979). In recent decades, a consensus has emerged that the syntactic structure of natural languages is well characterized in terms of the mildly context-sensitive hierarchy of languages (Weir 1988; Joshi, Shanker, and Weir 1991; Michaelis 1998; Kuhlmann 2013), a complexity class lying between context-free and context-sensitive and characterized by formal restrictions on various kinds of discontinuity in constituents. In dependency frameworks, these discontinuous constituents correspond to crossing dependencies (see Figure 1 for a simple example). Therefore, formal restrictions on discontinuous constituents correspond to formal restrictions on crossing dependencies in the ordered dependency tree (Kuhlmann 2013).

A number of formal restrictions on crossing dependencies have been proposed in the last 20 years, going beyond the simple observation that crossing dependencies are rare (Havelka 2007; Ferrer-i-Cancho, Gómez-Rodríguez, and Esteban 2018). We call these formal constraints on crossing dependencies crossing constraints. For example, Kuhlmann (2013) has proposed that dependency trees have limited gap degree and are usually well-nested (see Figure 2b). Pitler, Kannan, and Marcus (2013) propose that crossing dependency configurations have a property called 1-end-point-crossing. Other formal restrictions such as edge degree, multiplanarity, and heads’ depth difference have also been proposed (Yli-Jyrä 2003; Kuhlmann and Nivre 2006; Nivre 2007; Yadav, Vaidya, and Husain 2017). Among these crossing constraints, the constraint on gap degree is especially interesting, because gap degree defines the position of a formal language in the mildly context-sensitive hierarchy: A mildly context-sensitive language is defined by a finite upper bound on gap degree, with gap degree zero corresponding to a context-free grammar (Kuhlmann 2013; Marcus 1965).

The empirical arguments that crossing dependencies are constrained by factors such as gap-degree, and so forth, have typically come from demonstrations that crossing dependencies in a large number of observed trees in dependency corpora violate the constraints never or rarely (e.g., Kuhlmann and Nivre 2006; Havelka 2007). For example, Kuhlmann and Nivre (2006) show that only 0.17% of analyses in a Danish Dependency Treebank have gap-degree 2 and 99% of all non-projective structures are well-nested. These findings do not rule out the possibility that crossing constraints might manifest as epiphenomena of other, more general properties of dependency trees.

To appreciate this possibility, consider gap degree as an example. Gap degree is defined as the number of discontinuities in the projection of a node (see Figure 2 for an example), so it is upper-bounded by the number of discontinuities in the sentence. Given that crossing dependencies / discontinuous constituents are rare (Ferrer-i-Cancho, Gómez-Rodríguez, and Esteban 2018), we might expect to measure a low gap degree even if there is no true causally active constraint against gap degree.

This observation opens the possibility that crossing constraints such as gap degree, edge degree, and so on, could be epiphenomena of generic graph-theoretic properties of dependency trees, such as their height, arity, and so forth, and that together these generic factors drive the observable formal properties of crossing dependencies in natural language. If true, this would mean that apparent bounds on gap degree are accidental, and that formalisms such as mildly context-sensitive grammars fit linguistic data only because the structures that would violate them are rare by chance. It is this possibility that we explore in depth in Section 2.

Formal constraints such as mild context-sensitivity are usually associated with a competence-based approach to defining the generative capacity of language, where it is posited that there are (possibly innate) formal constraints on possible mental grammars. Such an approach can be evaluated against a performance-based approach. For example, a well-known cross-linguistic phenomenon termed syntactic islands is typically explained via constraints on structural configurations (Chomsky 1981). Broadly construed, “islands” prohibit formation of certain crossing dependencies and hence are important in circumscribing the formal domain of natural language. Notwithstanding the competence-based explanation, it has been argued that island constraints arise not due to structural ill-formedness, but rather due to constraints on human online language processing (e.g., Hofmeister and Sag 2010).

In a similar vein, the distribution of crossing dependencies in natural languages could also be understood in terms of processing-related constraints. It is a well known fact that, cross-linguistically, simple linguistic codes are more frequent, while complex linguistic codes are rare (e.g., Zipf 1949; Mahowald et al. 2013; Piantadosi 2014; Ferrer-i-Cancho et al. 2013; Ferrer-i-Cancho, Bentz, and Seguin 2020). Such a pattern has been argued to highlight the communicative efficiency of natural language (Jaeger and Tily 2011; Gibson et al. 2019). On this account, the restrictions on crossing dependencies in natural language could arise because such syntactic configurations are difficult (but not impossible) to produce and comprehend (Bresnan et al. 1982; Ades and Steedman 1982; Bach, Brown, and Marslen-wilson 1986; Joshi 1990; Ferrer-i-Cancho 2006,2014; Gómez-Rodríguez and Ferrer-i-Cancho 2017; Gómez-Rodríguez, Christiansen, and Ferrer-i-Cancho 2019). For example, given the incremental nature of language production (e.g., Ferreira and Henderson 1998), it could be assumed that production of crossing dependencies, which necessarily involves a discontinuity in a phrasal boundary, would incur an increased processing cost compared with non-crossing dependencies. Although there has been some experimental work on investigating the processing cost incurred during comprehension of sentences with crossing dependencies (e.g., Levy et al. 2012; Staub et al. 2018), a corpus-based empirical study investigating the influence of psycholinguistic factors beyond dependency distance on crossing dependencies is lacking. If certain psycholinguistic factors (e.g., working-memory constraints) can explain the occurrence of crossing dependencies, that could suggest a functional motivation of why such dependencies are rare in natural language.

In this article, we conduct a cross-linguistic corpus investigation into crossing dependencies both from the perspective of formal crossing constraints as well as from the perspective of processing constraints. In the first set of experiments we investigate if well-known crossing constraints (e.g., gap-degree, edge-degree) can account for crossing dependencies attested in various dependency treebanks. In particular, Section 2 compares the formal properties of crossing dependencies in real dependency trees with various random baselines matched in the number of crossing dependencies and other global graph-theoretic factors. In the second set of experiments, we conduct preliminary analyses to investigate whether certain psycholinguistic factors can account for crossing dependencies attested in various dependency treebanks. In particular, Section 3 models the tendency of two dependency arcs to cross given certain psycholinguistic metrics that are computed using the local configuration of the two arcs.

In this section we investigate evidence for global graph-theoretic constraints on crossing dependencies originating from the literatures on formal language theory and dependency parsing. Our goal is to determine which, if any, of these constraints really distinguish natural language dependency trees from the space of all possible directed trees. To do so, we study how often these contraints are violated in (1) real dependency trees and (2) a collection of random baseline trees controlling various properties of trees.

Chomsky (1956, 1957) first posed the question of how to characterize grammars of natural languages as computational objects, launching a research program that has attempted to describe sets of grammatical sentences using tools from mathematical logic, graph theory, and automaton theory. Since the late 1980s, a consensus has emerged that natural language is well characterized as falling within the mildly context-sensitive class of languages, a formal language class that is larger than the context-free languages, yet without taking advantage of the full expressive power of context-sensitive languages (Weir 1988; Joshi, Shanker, and Weir 1991).

These mildly context-sensitive languages are defined by constraints which turn out to be equivalent to constraints on crossing dependencies. Therefore, by answering the question of what constrains crossing dependencies, we can make progress toward understanding human languages at a computational level.

Crossing dependencies are related to formal language theory because they correspond to displacement phenomena in languages—structures that cannot be captured by a context-free grammar. Across grammatical formalisms, displacement phenomena and are modeled using a distinct kind of structure from non-crossing dependencies. Displacement phenomena (encompassing both extraposition and wh-dependencies) have been modeled in various ways in different syntactic frameworks:

• In the Minimalist tradition, non-crossing dependencies correspond to structures that can be built by the computational operation merge (or “external merge”), while crossing dependencies arise from the action of a distinct structure-building operation move (or “internal merge”) (Chomsky 1995; Stabler 1997; Michaelis 1998). A grammar with only merge would generate context-free languages and projective dependency trees.

• In phrase structure-based frameworks such as Lexical Functional Grammar (Bresnan 1982), Head-driven Phrase Structure Grammar (Pollard and Sag 1994), and Combinatory Categorial Grammar (Steedman and Baldridge 2011), displacement phenomena are handled using phrase structure rules defined in a way that allows information to percolate through a tree in a non-local manner, a mechanism called “slash-passing.”

• Some theories of dependency grammar invoke the idea that each word in a sentence has both a syntactic “head” and a syntactic “governor,” which may coincide. Arcs drawn from governors to dependents may cross, but arcs drawn from heads to dependents never cross (Groß and Osborne 2009). Crossing dependencies correspond to cases where the syntactic governor is distinct from the syntactic head. What we are calling “heads” in this article would be “governors” in such theories.

These various formalisms allow for crossing dependencies by different mechanisms, yet with constraints which turn out to be similar or equivalent across formalisms. In particular, most mildly context-sensitive formalisms end up instantiating bounds on a quantity called gap degree. This quantity goes by different names depending on the grammar formalism. It is equivalent (up to additive constants) to block degree or fan-out in linear context-free rewriting systems (Kuhlmann 2007, 2013), the number of components in multiple context-free grammars (Seki et al. 1991), the maximal rank of a coupled context-free grammar (Hotz and Pitsch 1996), the number of licensee features in Minimalist Grammars (Michaelis 1998; Boston, Hale, and Kuhlmann 2010), and others. It was first introduced in a dependency framework by Holan et al. (1998), and shown to relate to mild context-sensitivity by Kuhlmann (2007). Some mildly context-sensitive formalisms also induce a constraint called well-nestedness, which can also be reduced to constraints on crossing dependencies (Bodirsky, Kuhlmann, and Möhl 2005).

Crossing constraints have also been of interest for those studying the development of efficient dependency parsing algorithms. Such algorithms are generally only available for trees with constrained crossings. For example, if we assume that all trees are projective, then we can perform exact parsing in time cubic in the sentence length (O(n³)) by reducing the dependency grammar to a lexicalized context-free grammar (Eisner and Satta 1999). If we assume all trees are well-nested and gap degree is bounded, then we can generally parse in polynomial time. Without the constraint of well-nestedness, parsing becomes NP-hard (Satta 1992; Gómez-Rodríguez, Carroll, and Weir 2011).

The parsing literature has also been the source of a number of new formal constraints on crossing dependencies, beyond those introduced in the formal syntax literature. For example, Pitler, Kannan, and Marcus (2013) propose a constraint called 1-end-point-crossing. If we assume that all dependency trees are 1-end-point-crossing, then we can parse in quartic time (O(n⁴)) (see also Gómez-Rodríguez, Shi, and Lee 2018).

Our goal is to determine if there is really evidence for formal graph-theoretic crossing constraints on crossing dependencies in depedency treebanks beyond what can be explained in terms of more generic properties of dependency trees. Below, we list and define the formal crossing constraints that we test. In our terminology we strive to follow Kuhlmann and Nivre (2006).

The projection of a node X is the ordered list of all the nodes transitively dominated by X plus X itself. For example, in the dependency tree in Figure 2, $Xg,Xd,Xh$ is the projection of the node X_h. A projection is discontinuous if it forms a discontinuous substring of the sentence. For example, the projection of X_h has two discontinuities, one between X_d and X_h, and another between X_g and X_d. The gap degree of a tree is the largest number of discontinuities in the projection of any node in the tree.

The subtree rooted at a node X is the set of all the transitive nodes dominated by X plus X itself. For example, in the dependency tree (a) and (b) in Figure 3, {X_a,X_b,X_e} is the subtree rooted at node X_e, and {X_c,X_d} is the subtree rooted at node X_d. Two subtrees with nodes {P,Q} and {R,S} interleave if the nodes are in linear order such that P < R < Q < S. A dependency tree is ill-nested if and only if two of its disjoint subtrees interleave. For example, in (3a), {X_a,X_b,X_e} and {X_c,X_d} are two disjoint subtrees but they do not interleave as the nodes are in the order X_a < X_b < X_c < X_d < X_e. Therefore, tree (3a) is well-nested. In (3b), the disjoint subtrees {X_a,X_b,X_e} and {X_c,X_d} interleave as the order of the nodes is X_a < X_c < X_b < X_d < X_e. The dashed red arc creates the ill-nestedness. Ill-nestedness implies gap degree >1.

Let e be the span of dependency arc $Xh→Xd$⁠. The span e consists of nodes between a head X_h and its dependent X_d, which are X_i, X_a, and X_b in Figure 4. The edge degree of a dependency arc $Xh→Xd$ is the number of nodes in the span e that are neither transitively dominated by some node in the span e nor transitively dominated by the head X_h (Kuhlmann and Nivre 2006). For example, the arc $Xh→Xd$ in Figure 4a and 4b has an edge degree of 2 because nodes X_i and X_b are not dominated by any node in the span e. In addition, they are also not dominated by the head X_h. The edge degree of a dependency tree is the highest edge degree among the arcs of the tree.

There are cognitive reasons to suspect that edge degree might be limited in natural language. From an online processing perspective, higher edge degree in a subtree results in a need to maintain an unresolved crossing dependency across a longer span of words, which may result in online processing difficulty due to higher working memory load (Gibson 1998).

The end-point crossings of a dependency arc is the number of distinct heads of all edges that cross the arc. More formally, given an arc $Xh→Xd$ with a span e, the end-point crossings of arc $Xh→Xd$ is defined as the number of distinct heads of nodes in e that are not part of the projection of X_h nor of any element of e. The end-point crossings of a tree is the maximum end-point crossings of any arc in the tree. For example, in Figure 4a, the number of heads modified by X_i and X_b is 1 (corresponding to X_j), therefore, the end-point crossing is 1. In Figure 4b, the number of heads modified by X_i and X_b are 2 (corresponding to X_j and X_r, respectively); therefore, the end-point crossing is 2.

It has been argued that natural language dependency trees tend to have not more than one end-point crossing, which is called the 1-end-point-crossing constraint. Pitler, Kannan, and Marcus (2013) argue that this constraint is related to the Phase Impenetrability Condition from Minimalist syntax (Chomsky 2007). From a processing based perspective, higher end-point crossings in a subtree should lead to multiple heads/dependents being maintained/stored at the same time in the parse stack. This should lead to increased storage cost (Gibson 1998). In addition, a longer span of the crossing dependency could lead to similarity-based interference (Lewis and Vasishth 2005) at the head.

In terms of formal syntax, HDD can correspond to the hierarchical depth between a filler and a gap in a long distance dependency (e.g., wh movement). Based on the theoretical syntax literature, HDD should be unbounded, at least for leftward wh-dependencies (Sag, Wasow, and Bender 1999). However, increasing HDD seems to correlate with increased online processing difficulty for humans (Phillips, Kazanina, and Abada 2005). More generally, HDD has been proposed (see Yadav, Vaidya, and Husain 2017) to formalize the experimental findings that increased embedding depth leads to processing difficulty (e.g., Yngve 1960; Gibson and Thomas 1999). Therefore, it is possible that HDD is restricted in dependency trees due to cognitive constraints.

In order to demonstrate our crossing constraints, Figure 6 shows an example of a complex dependency tree in Hungarian, drawn from the Szeged Dependency Treebank. The tree has 3 crossing dependencies, with gap degree 2: The projection from kiejteni has two discontinuities. The tree has edge degree 2, end-point crossings 2, and HDD 2. It has sentence length 10, arity¹ 2, and tree height² 6.

To test if crossing constraints (e.g., gap degree, edge degree) can account for crossing dependencies attested in natural language treebanks, we compare the distribution of crossing constraints in natural languages with the random baselines matched in number of crossing dependencies and other tree properties. For example, to test if gap degree is a constraint—over and above the constraint on number of crossings and dependency lengths—we compare the distribution of gap degree in natural languages with the random baseline matched in number of crossings and dependency lengths. We describe the random baselines and statistical method used for comparison next (cf. Yadav, Husain, and Futrell 2019).

We use four random baselines to assess whether crossing constraints occur independent of rate of crossing dependencies and other tree properties like tree height or dependency lengths in natural languages. They include random trees, random linear arrangements, dependency length (DL)-controlled random trees, and DL-controlled RLAs. Each baseline controls a particular set of tree properties as shown in Table 1.

In order to generate a random tree corresponding to a real language tree of sentence length n, we first sample from a uniform distribution over tree structures with n nodes using Prüfer codes (Prüfer 1918), and then use rejection sampling to obtain a tree that matches the real tree in the number of crossing dependencies. The resulting distribution is uniform over all tree structures with the specified length and number of crossing edges. A DL-controlled random tree must match a real tree both in terms of its number of crossings and in terms of the distribution of dependency lengths within the tree (for details, see Yadav, Husain, and Futrell 2021). This procedure samples from the uniform distribution on trees with specified length, number of crossings, and distribution of dependency lengths.

Random linear arrangements (RLAs) are generated by permuting the linear order of nodes in a real languages tree. A reordered tree that matches in number of crossings with a real tree is accepted as a valid sample for RLAs. To generate DL-controlled RLAs, we sample from RLAs that match in the distribution of dependency lengths with the real trees.

These baseline trees are all generated by a rejection sampling procedure that rejects the vast majority of samples. As such, it is only possible to generate sentences with length up to 11 using this method given currently available computing resources.

All baselines studied in this article control for the number of crossings per sentences. For baselines that control for dependency lengths but not the number of crossings, see Yadav, Husain, and Futrell (2021).

For natural languages data, we use treebanks from Surface-syntactic Universal Dependencies (SUD) v.2.4 (Gerdes et al. 2018, 2019). We test on treebanks from 56 languages, excluding treebanks with less than 500 sentences and ancient languages.

The data used for the analysis contained 9 head-final and 47 head-initial languages for the analysis, as determined by Yadav et al. (2020), which focuses on verb–object relations. The head-final languages were: Afrikaans, Dutch, German, Hindi, Japanese, Korean, Persian, Tamil, and Urdu. The head-initial languages were: Amharic, Arabic, Bulgarian, Bambara, Catalan, Czech, Danish, Greek, English, Spanish, Estonian, Basque, Finnish, Faroese, French, Irish, Galician, Hebrew, Croatian, Upper Sorbian, Hungarian, Armenian, Indonesian, Italian, Kazakh, Northern Kurdish, Lithuanian, Latvian, Maltese, Erzya, Norwegian, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Northern Sami, Serbian, Swedish, Thai, Turkish, Uyghur, Ukrainian, Vietnamese, Wolof, and Chinese.

The SUD treebanks have been converted from the Universal Dependencies (UD) treebanks (Nivre 2015) to reflect syntactic dependencies rather than the more semantic dependencies favored by UD. We also did an additional analysis on the corresponding UD treebanks. We found that the rate of violating crossing constraints is lower in UD trees but the overall pattern of results is the same as for SUD treebanks with two exceptions: (1) the constraint on end-point crossings receives weaker evidence, and (2) the constraint on well-nestedness receives stronger evidence. Regarding point (1), this is likely due to UD’s flat structure, which means that many nodes share a head. See supplementary materials S1 for a comparison of results obtained from SUD vs. UD data.

We believe the SUD analysis is most appropriate because SUD reflects the syntactic relationships and analyses that were used in the development of theories of formal crossing constraints (Joshi 1985), including early work investigating dependency treebanks (Nivre 2006).

To test whether real language trees differ from random baseline trees in the distribution of crossing constraints, we fit mixed-effect Poisson regression models. Poisson regression is appropriate for modeling nonnegative integer-valued data such as the formal crossing properties. The dependent variables in the models are the rate of violations of crossing constraints (gap degree, edge degree, etc.).

To evaluate evidence for the hypothesis whether gap degree is lower in real trees compared with baseline trees, we compare the model in Equation (2) with a null model that lacks the R_ij term, using a likelihood ratio test. We report log-likelihood ratio values, interpreting them as strength of evidence for a difference between real and random trees.

The log-likelihood ratios can be interpreted as logarithmic Bayes factors comparing two hypotheses with equal prior probability: H₀, that there is no distinction between real and random trees; and H₁, that there is a distinction as given by the regression coefficients β₂ and β₃. A higher log-likelihood ratio indicates stronger evidence for H₁.

The distributions of the formal measures in real trees and random baselines are shown in Figure 7. Table 2 summarizes results in terms of log-likelihood ratios. We find that there is uniformly strong evidence that edge degree, end-point crossings, and HDD are different between real and random trees. This means that the distribution of these formal properties cannot be explained solely in terms of generic constraints on number of crossings, tree topology, and dependency length.

However, for gap degree and well-nestedness, the picture is different. When comparing real trees against DL-controlled random linear arrangements, we do not find substantial evidence for differences in well-nestedness or gap degree. When comparing against random linear arrangements, we do not find substantial evidence for differences in gap degree. These log-likelihood ratios are dramatically smaller than those for, for example, HDD. If there is a distinction in gap degree between real and random trees, the evidence for this distinction is dramatically small compared with other constraints.

We tested whether there is evidence for graph-theoretic constraints on crossing dependencies beyond what can be explained in terms of other, generic properties of natural language trees.

We find two key results:

1. There is decisive evidence that edge degree, end-point crossings, and HDD are different between real and random trees, suggesting that any constraints on these quantities cannot be explained merely in terms of the number of crossings, tree topology, and dependency length distribution in natural language trees.

2. We find insubstantial evidence for gap degree and well-nestedness constraint after controlling for the number of crossings, tree topology, and dependency length.

Taken together, these results suggest that, despite the massive literature on gap degree and well-nestedness and their connections to formal language theory, these constraints are not the formal properties that most strongly characterize crossing dependencies. In comparison, edge degree, end-point crossings, and HDD—measures that have emerged from the parsing literature, not the formal language theory literature—emerge as strongly characteristic of crossing dependencies. We additionally note an important caveat that our results hold only in short trees, and might be different in larger trees (cf. Ferrer-i-Cancho et al. 2021). The patterns for longer sentences can only be confirmed by baseline generation, which will be taken up in future work.

Among the crossing constraints investigated in the current work, the evidence for HDD as a crossing constraint is strongest. Recall that the HDD constraint is motivated by the findings in the psycholinguistic literature that increased embedding leads to processing difficulty. This suggests that constraints on crossing dependencies could be driven by processing considerations. We turn next to a preliminary investigation where this possibility is explored further.

In the previous section we investigated the role of formal graph-theoretic factors in determining the distribution of crossing dependencies in natural language. As stated earlier, these factors can be construed as capturing the competence-based constraints on grammar. However, natural language grammar can, in principle, also be influenced by processing-based constraints. One such early proposal can be found in Joshi (1985), where efficient (asymptotic) parsing complexity is a key design requirement for natural languages. More recent proposals include the role of efficient parsing in shaping word order in natural language (see, e.g., Hawkins 2004, although his notion of “efficiency” is quite different from Joshi’s). On this account, rarity of crossing dependencies across languages could be assumed to reflect processing difficulty in handling such configurations (also see Ferrer-i-Cancho 2014).

Given the incremental nature of language production (e.g., Ferreira and Henderson 1998), it is reasonable to assume that production of crossing dependencies, which necessarily involves a discontinuity in a phrasal boundary, would incur an increased processing cost compared with non-crossing dependencies. Indeed, recent work investigating filler–gap dependencies (Momma 2021) suggests that such crossing dependencies might require additional cognitive resources in planning. Given the evidence for the tight link between production and comprehension difficulty (MacDonald 2013; Scontras, Badecker, and Fedorenko 2017), it is therefore not far-fetched to assume that comprehenders should also find crossing dependencies difficult to process (Levy et al. 2012; Yadav, Vaidya, and Husain 2017; Husain and Vasishth 2015; Staub et al. 2018). Indeed, it is well known that, given an unbounded dependency, the comprehension system tries to resolve it as soon as possible, a constraint known as active-filler strategy (Frazier 1987). The experiments discussed in this section, therefore, explore whether processing factors such as working-memory constraints, predictability, and so on, could play a role in determining the occurrence of a crossing dependency in the input.

The key motivation of this preliminary study is to investigate if processing related factors modulate the occurrence of a crossing dependency. We investigate two factors—namely, working-memory constraints and prediction processes.

The influence of working-memory limitations on sentence comprehension as well as production is well attested. Cross-linguistically, the linear distance between syntactically related words (i.e., dependency length) has been found to be minimized, in the phenomenon of dependency length minimization (for reviews, see Liu, Xu, and Liang 2017; Temperley and Gildea 2018). This dependency length minimization has recently been argued to be a manifestation of information locality (Futrell 2019; Hahn, Degen, and Futrell 2021), the key idea being that words with high pointwise mutual information (PMI) tend to be close to each other. The “closeness” between a pair of words is usually operationalized as linear distance between them, but more generally “closeness” would mean the simplicity of the structure that intervene a pair of words (see Yadav, Mittal, and Husain 2020). This generalized version of information locality can be termed as information-simplicity: The words with high PMI tend to have simpler intervening structure between them.

Together, dependency length minimization and information-simplicity have clear implications for crossing dependencies. First, the information-simplicity hypothesis predicts that a head-dependent pair with high PMI is less likely to be involved in a crossing dependency. Second, dependency length minimization implies that long phrases could be extraposed in order to avoid increased dependency distance in situ. We call this heavy-phrase extraposition. The heavy-phrase extraposition hypothesis predicts that dependencies with a heavy dependent, in terms of length of the phrase, are more likely to be involved in a crossing dependency. Additionally, we expect a positive correlation between dependency length and crossing tendency: The dependencies which are shorter in length are less likely involved in a crossing configuration (Ferrer-i-Cancho 2014; Ferrer-i-Cancho and Gómez-Rodríguez 2016). We call this localized-simplicity hypothesis: Localized words have simpler intervening structures between them.

Apart from the role of working memory, another factor that has garnered much attention in the processing literature is prediction. Sentence processing is known to involve a robust top–down component that involves a preactivation of upcoming linguistic material. Such predictions are known to facilitate comprehension (Levy 2008; Smith and Levy 2013) and to attenuate the cost of memory constraints (Husain, Vasishth, and Srinivasan 2014). On this account, when the presence of a dependent is highly expected, this high expectation could offset any cost incurred due to crossing dependency (Levy et al. 2012). So, we test the expectation hypothesis that a crossing configuration is more likely in situations where the upcoming dependent/head is highly expected. Expectation of a dependent given a head is operationalized as the log probability that a head has at least one outgoing dependency with that relation type.

Given the gap between the point at which prediction is made and the point at which the concerned linguistic entity is received via input, linguistic predictions have to be maintained in memory (Gibson 1998). This maintenance cost has been shown to correspond to measurable processing difficulty (Husain, Vasishth, and Srinivasan 2015; Ristic et al. 2021), that is, prediction maintenance over a longer period can be costly. It is therefore expected that longer maintenance of dependencies involved in crossing should be avoided. In this work we operationalize maintenance as (a) the number of words between the heads of the crossing dependencies, and (b) the number of heads between the heads of the crossing dependencies.³ We expect that dependencies that involve a crossing configuration tend to have shorter distance between their heads both in terms of number of words and number of heads. We call this head-head locality hypothesis: Two dependencies with their heads being far away from each other are less likely to form a crossing configuration.

In order to assess the role of various processing factors mentioned above, we fit logistic regression models to predict whether two dependencies cross.

Given dependency i and dependency j in a sentence (see Figure 8), this model uses the following predictors to predict whether dependency i is crossed by dependency j or not. We call i a host dependency and j a crosser dependency.

1. Head-dependent pointwise mutual information of dependency i and dependency j, i.e., PMI_i and PMI_j

2. Expectation of seeing a dependency relation given head type for dependency i and dependency j, i.e., EXP_i and EXP_j

3. Distance between head and dependent of dependency i and dependency j, i.e., DD_i and DD_j

4. Weight of dependent of dependency i, W_i, i.e., the number of nodes transitively dominated by the dependent of i plus one

5. Linear distance between head of dependency i and head of dependency j, LHD

6. Hierarchical distance (number of heads) between head of dependency i and head of dependency j, HHD

Given the discussion in Section 3.1, we predict the following:

• The information-simplicity hypothesis predicts that a dependency with high pointwise mutual information is less likely to get involved in a crossing construction. Thus, it predicts a negative estimate for the parameters β₁ and β₅.

• The heavy-phrase extraposition hypothesis predicts that if a dependency has a heavy dependent, it is more likely to get crossed. Thus, a positive estimate is predicted for β₄.

• The localized-simplicity hypothesis predicts a positive correlation between dependency length and crossing tendency, that is, a positive estimate for β₃ and β₇.

• The expectation hypothesis predicts that a dependency with high expectation is more likely to involve a crossing construction, namely, a positive estimate for β₂ and β₆.

• Finally, the head-head locality hypothesis predicts that two dependencies with longer distance between their heads (i.e., higher LHD or HHD) are less likely to cross each other, that is, β₈ and β₉ should show a negative estimate.

We choose 12 languages from the Surface-syntactic Universal Dependencies (SUD v2.4) treebank, out of which 7 are head-initial and 5 are head-final languages. The criteria for language selection is based on corpus size ( >5,000 trees) and language typology. Out of the 7 languages, Hindi, Dutch, Japanese, German, and Korean are head-final languages; the remaining languages, English, Arabic, French, Spanish, Italian, Polish, and Romanian, are head-initial.

Table 3 summarizes results from the logistic regression model. Results show that all the predictors, namely, pointwise mutual information, expectation, dependency length, dependent weight, distance between the head of the host, and head of the crosser dependency, have significant effect on crossings in the expected directions.

We find an effect of pointwise mutual information for most of the languages such that two dependencies tend to cross each other if their head-dependent mutual information is low. In other words, a dependency that has relatively low mutual information between its head and dependent is more likely to cross—and get crossed by—another dependency. The result supports the information-simplicity hypothesis.

Additionally, all 12 languages show an effect of the dependent weight such that a dependency with heavier dependent phrase is more likely to get crossed by another dependency. The result supports the heavy-phrase extraposition hypothesis.

The results also support the localized-simplicity hypothesis, that is, dependency distance is positively correlated with crossing tendency—a shorter dependency is less likely to be involved in a crossing construction.

With regard to the role of prediction, results show that a dependency relation that has relatively low expectation given a head type is more likely to cross—and get crossed by—another dependency. The result does not support the expectation hypothesis. Finally, we find that the linear and hierarchical distance between the heads of two dependencies negatively influence whether they cross or not. Crossing tendency reduces if the number of heads/words between the heads of the two dependencies increases. This result holds for most of the languages (except Hindi, Spanish, Dutch, and Polish). The LHD/HHD results support the head-head locality hypothesis motivated by prediction maintenance account.

We also test a prediction by Levy et al. (2012) that a crossing dependency is easier to comprehend if expectation of a dependency relation given a nominal head is higher. We test whether expectation has a positive effect on crossing tendency of noun-headed constructions (see Figure 9). In addition to expectation effect, we also test the effect of weight of dependent of noun phrase—a noun-headed dependency is more likely to get crossed if its dependency is heavier because the heavy dependent may move around to minimize dependency length. To test the hypothesis, we fit a logistic regression similar to 3 with two predictors, namely, expectation of dependency relation and dependent weight.

Table 4 shows the estimates of effect of expectation and dependent weight on crossing tendency of a noun-headed construction. The effect of expectation has a negative estimate (except Japanese), suggesting that if expectation of seeing a particular dependency relation in a noun-headed dependency is higher, the dependency is less likely to get crossed by another dependency.

The weight of dependent in a noun-headed construction positively affects its crossing tendency (except Japanese)—a noun-headed construction has a higher tendency to get crossed by another dependency if the weight of its dependent is higher. The result supports the heavy-phrase extraposition hypothesis: A heavy dependent is more likely to get extraposed—hence causing a crossing dependency—in order to minimize dependency distance.⁴

We also find a significant interaction effect between dependent weight and expectation such that effect of dependent weight increases with increase in expectation. This result suggests that high expectation facilitates crossing dependencies with heavy dependent. In other words, a heavy dependent is more likely to get extraposed when expectation is high. This pattern has some support for the hypothesis proposed by Levy et al. (2012) that increased expectation facilitates the processing of right-extraposed structures.

One concern regarding the significant effect of formal measures like dependency distance, dependent weight, and inter-head distance could be that they do not capture any linguistic or cognitive phenomenon, rather, their effect could be due to independent reasons such as tree topology. For example, it is possible that an arc in any random directed acyclic graph is more likely to get crossed by another arc if linear distance between its nodes is larger (Ferrer-i-Cancho and Gómez-Rodríguez 2016).

To check whether some natural language phenomena underlies the effects of dependency distance, dependent weight, and so forth, on crossing dependency, we fit a model to predict crossing tendency in real vs. random trees. The model includes interaction effect terms to test whether the expected effect of dependency distance, dependent weight, and so on, is larger in real trees compared with the random trees generated as part of Section 2. If the effect of these predictors arise due to some real psycholinguistic phenomena, we expect a positive estimate for the interaction effect.

Table 5 shows the results of the model. We find a significant interaction effect for all the predictors.

• The effect of dependency distance on crossing tendency is larger in real language trees, such that longer dependencies are more likely to cross or get crossed by another dependency in real trees compared to random trees.

• The effect of dependent weight is driven by real trees, such that a dependency with heavy dependent is more likely to get crossed in real trees, but not in random trees (the main effect is in opposite direction).

The results from the logistic regression models provide compelling evidence for the role of working-memory constraints and information-locality on the occurrence crossing dependencies. To summarize, we find support for:

1. Information simplicity: The words with high mutual information are less likely to form crossing configurations.

2. Localized simplicity: The words that are close to each other are less likely to involve a crossing dependency.

3. Heavy-phrase extraposition: A dependency with a heavy dependent is more likely to get crossed by another dependency.

4. Head-head locality: Two dependencies with their heads being far away from each other are less likely to form a crossing configuration.

However, the results were only partly consistent with the expectation hypothesis: High expectation for the presence of a dependent correlates with a higher rate of crossing only when that dependent is heavy, as indicated by the presence of a positive interaction of weight and expectation in Table 5 for most languages, but a negative main effect of expectation. Overall, we found evidence for working-memory based and information-locality based accounts, but limited evidence for expectation-based accounts.

An interesting result that should be interpreted in the light of previous findings is the effect of dependency lengths on the occurrence of crossing dependencies. Recent work has shown that a constraint on dependency length alone cannot fully explain the low rate of crossing dependencies in natural languages (Yadav, Husain, and Futrell 2021). In contrast, our results show an effect of dependency length on crossing tendency such that longer dependencies are more likely to cross—or get crossed by—other dependencies. Additionally, our results in Section 2 show that dependency lengths can explain ill-nestedness distribution in natural languages. Together, these findings suggest that although dependency length affects the occurrence of crossing dependencies, it cannot fully explain the observed quantitative distribution of crossings in natural languages.

The current work investigated crossing dependencies from two perspectives. These were (a) the role of certain global graph-theoretic factors in determining the distribution of crossing dependencies in natural language, and (b) the role of certain local psycholinguistic factors in determining if two dependencies cross. Results from Experiment 1 provide strong evidence that edge-degree, end-point crossings, and HDD determine the distribution of crossing dependencies in natural language, while they provide weak to insubstantial evidence that other factors such as gap-degree and well-nestedness constrain crossing dependencies. This suggests that the apparent bounds on gap-degree and well-nestedness arise as a consequence of factors such as number of crossings, tree topology, and dependency length.

In particular, the results from Experiment 1 provide the strongest evidence in favor of cognitively motivated crossing constraints (such as HDD), suggesting a parsing/processing driven constraint on crossing dependencies. This proposal was further substantiated through the results from Experiment 2 that show that certain psycholinguistically motivated factors such as information locality, dependency weight, as well as inter-head distance can determine if a pair of dependency will cross, even when compared against random trees. These results provide evidence for a functional motivation for distribution of crossing dependencies in natural language. The key contribution of this work is to (a) provide a method to quantify the evidence for a particular graph theoretic constraint beyond what can be explained in terms of tree topology, number of crossings, and dependency distance, and (b) to highlight that crossings can be predicted through processing factors beyond dependency distance.

Together, the two findings suggest that graph-theoretic constraints on crossing dependencies could be driven by processing considerations (also see Ferrer-i-Cancho and Gómez-Rodríguez 2016). The idea that formal properties in grammar could be determined by processing consideration is not new. As stated earlier, Joshi (1985) proposed the MCS hypothesis, which required grammars to be efficiently parseable, in the sense of worst-case asymptotic complexity of exact parsing. Similarly, the performance-grammar correspondence hypothesis by Hawkins (2004) proposes that processing strategies get grammaticalized for efficiency considerations. We note that, on these accounts, efficiency in grammar is understood as efficiency in online comprehension of the utterances licensed by the grammar.

There is evidence that humans find crossing dependencies to be difficult during comprehension. One piece of evidence for this comes from processing of filler–gap dependencies. A filler-gap dependency typically involves a crossing dependency and, in order to resolve it, the human processing system is known to operate with the principle of immediacy, namely, the parser tries to resolve it sooner rather than later (De Vincenzi 1991). There is also evidence that, if possible, such dependencies are avoided (Staub et al. 2018). More recently, Husain and Yadav (2020) show that in Hindi, crossing dependencies are avoided during comprehension of participle clauses and that such dependencies lead to processing difficulty. Processing of crossing dependency necessarily involves maintenance of unresolved structure and a processing strategy to avoid building such complex configuration is consistent with the parser’s bias for building simple structures (Frazier 1985). In the domain of computational parsers, limits on quantities such as edge-degree, end-point crossings, and so forth, have been previously shown to lead to efficiency in terms of both asymptotic complexity and practical accuracy (Pitler, Kannan, and Marcus 2013), but there has not yet been any systematic investigation on whether these factors affect human online processing. If it is true that constraints on crossing dependencies are motivated by processing efficiency for humans, then formal factors such as end-point crossings might also lead to observable processing difficulty for humans.

The research discussed above suggests that the distribution of crossing dependencies may arise due to comprehension difficulty, but processing efficiency is not limited to comprehension only: Production and learning may also play a role. For example, it is known that difficulty during comprehension could be a consequence of pressures during production (MacDonald 2013); in this view, ease of production determines the distribution of linguistic patterns in a language community, and this distribution in turn makes certain structures either easy or difficult to comprehend. Learning may also play a role: Given that crossing dependencies are difficult to produce and comprehend, it is possible that such dependencies are also difficult to learn and that their rarity could be independently driven by learning biases (Chang 2009). Teasing apart the role (and extent) of how various processing factors affect crossing dependencies will require a dedicated research effort. We hope that our work is a step in that direction.

If crossing dependencies are difficult to process, then why do they exist at all? To answer this question, we need to appreciate that processing cost due to crossings could be just one of the many sources that lead to processing difficulty. Decades of research on comprehension and production has shown that processing complexity can be understood as a trade-off between various countervailing factors related to sentence encoding/decoding (e.g., Trueswell, Tanenhaus, and Garnsey 1994; Kaiser and Trueswell 2004; Altmann and Kamide 1999; Frazier 1979; Gibson 1998; Lewis and Vasishth 2005; Levy 2008). For example, syntactic configurations leading to clausal embeddings are known to be quite complex and under such configurations creating a crossing dependency (for example, by right-extraposition) could in fact lead to a relatively less complex structure (Yngve 1960). This, of course, implies that in certain context, other syntactic configurations could be costlier. Similarly, requirements at different levels of linguistic encoding (discourse, pragmatics, etc.) could require positioning of words that could lead to creation of crossing dependencies. Finally, during production, accessibility-related pressures (e.g., Ferreira and Dell 2000; Branigan, Pickering, and Tanaka 2008) could lead to creation of crossing dependencies. In short, formation of crossing dependencies could be due to (a) creation of less costly structures, (b) extra-sentential requirements, and (c) production pressures such as accessibility.

Could crossing dependencies arise due to non-functional reasons? It is possible that a language-wide grammatical constraint (e.g., on the position of heads) could lead to existence of certain crossing dependencies. For example, in English, the presence of wh crossing dependencies could be a consequence of a fixed word order in the language. Compare this to wh dependencies in a free word order language like Hindi where the wh phrase can appear in situ, thereby creating a non-crossing dependency. For such cases, avoiding a crossing dependency in a language like English is not possible due to language-wide grammatical constraints.

We hope this work stimulates future work studying the functional bases for restrictions on crossing dependencies using experimental and corpus methods.

We thank the three anonymous reviewers for helpful suggestions. This work was supported by an NVIDIA GPU grant to RF.