Joint Transition-Based Models for Morpho-Syntactic Parsing: Parsing Strategies for MRLs and a Case Study from Modern Hebrew

NLP research in recent years has shown increasing interest in parsing typologically different languages, as evident, for instance, by the universal dependencies¹ initiative Nivre et al. (2016). In particular, much attention is drawn to parsing morphologically rich languages (MRLs), which differ significantly from English in their structure and characteristics (Tsarfaty et al., 2010).

In MRLs, grammatical information, typically expressed using word order in English, is often manifested in the internally complex structure of the words. Words in MRLs may carry, in addition to lexical content, functional affixes and clitics that correspond to additional pieces of information. In Modern Hebrew, for example, the inflected verb “ahbtih”² (loved + 1pers.singular.past + 3pers.feminine.singular) corresponds to three different grammatical functions: the subject “I,” the predicate “loved,” and the direct object “her.” Similarly, Spanish dámelo corresponds to a predicate, an indirect object, and a direct object, as in “give it to me.” Thus, in MRLs, morphological analysis (MA) which translates raw space-delimited tokens to syntactically relevant “word” units is a necessary condition for any syntactic or semantic downstream task.

However, raw space-delimited tokens in MRLs are often highly ambiguous. In Hebrew, Arabic, and other Semitic languages, this situation is further complicated by fact that written texts lack diacritics. The Hebrew token “fmn,” for instance, may be read as the noun “oil,” the adjective “fat,” the verb “lubricated,” the sequence “that”+“of,” or the phrase “their”+“name,” only one of which is relevant in context. This has clear ramifications for dependency parsing. Figure 1 shows a lattice that captures all possible analyses of the Hebrew phrase “bclm hneim,” literally: “in-the-shadow-of-them the-pleasant,” translated “in their pleasant shadow.” Each lattice arc corresponds to a potential node in a dependency tree. Dark circles mark morpheme boundaries, double circles mark token boundaries. The top tree depicts a correct syntactic analysis. In the bottom tree, incorrectly disambiguated tokens lead to a wrong syntactic analysis.

Previous dependency parsing evaluation campaigns (Buchholz and Marsi, 2006; Nivre et al., 2007) assumed that the correct morphological analysis and disambiguation (MA&D) of the input stream is known in advance. In realistic end-to-end parsing scenarios, however, this is of course not so. To overcome this, pipeline architectures where MA&D precedes parsing have been set up. These pipelines are suboptimal since they suffer from error propagation, and since local linear context available for automatic MA&D may be insufficient for accurate morphological disambiguation. For this, actual syntactic context may be required (Tsarfaty, 2006). To resolve this apparent loop, where morphological analysis is required for syntactic parsing and syntactic analysis is required for morphological disambiguation, Tsarfaty (2006) hypothesised that joint morphosyntactic parsing, where morphological information may assist syntactic disambiguation and vice versa, may be better suited.

This joint morphosyntactic hypothesis has been taken up and successfully confirmed in the context of phrase–structure parsing for Semitic languages (Goldberg and Tsarfaty, 2008; Cohen and Smith, 2007; Green and Manning, 2010). For dependency parsing, Bohnet and Nivre (2012) and Bohnet et al. (2013) present language-agnostic transition-based frameworks for jointly parsing and tagging input words, though without addressing the complex issue of retokenizing ambiguous input tokens. More recently, Seeker and Centinoglu (2015) presented a graph-based framework for lattice parsing of Turkish also covering morphological segmentation. Their system takes a “product of experts” approach wherein the morphological paths and dependency trees are handled via two distinct models (a linear model over bigrams for MD and an arc-factor model for dependencies), reaching agreement via a dual decomposition setup.

In this work, we present a novel, language-agnostic, transition-based framework for end-to-end morphosyntactic dependency parsing. The framework unifies a morphological and a syntactic component into a joint parser encompassing a single transition system, a single objective function, joint learning, and joint decoding. We apply this system to parsing Modern Hebrew and empirically confirm that predicting MA&D in the joint settings improves upon standalone MA&D, and upon recently reported Hebrew MA&D results. Our system further improves end-to-end dependency parsing results in comparison to existing state-of-the-art parsers in pipeline scenarios, it significantly outperforms the joint parser of Seeker and Centinoglu (2015), and it substantially outperforms the dependency parser of Goldberg and Elhadad (2010), so far considered the de facto standard for Hebrew dependency parsing.

The contribution of this paper is thus threefold. First, we define a language-agnostic joint morphosyntactic parser in a transition-based framework. Secondly, we empirically confirm that MA&D benefits from syntactic parsing, and in realistic end-to-end parsing scenarios, also vice versa. Finally, we present a new set of strong Hebrew end-to-end parsing results and deliver an open-source, language agnostic implementation of the joint parser, for further investigating joint morphosyntactic parsing strategies. This paper is organized as follows. In Section 2, we present our formal framework (2.1), morphological model (2.2), syntactic model (2.3), and joint framework (2.4). Sections 3 and 4 present our experiments and analysis, respectively. Section 5 discusses related and future work, and Section 6 concludes.

We cast end-to-end morphosyntactic parsing as a structure prediction function $F:X→Y$⁠, where $x∈X$ is a sequence of raw input tokens and $y∈Y$ is a dependency representation where the nodes in the tree correspond to disambiguated morphosyntactic units we refer to as morphemes.³

We assume that F is realized in a transition-based framework augmented with the structure prediction method of Zhang and Clark (2011). We start off with a completely general definition of a transition system as a quadruple S = (C,T,c_s,C_t), with C a set of configurations, T a set of transitions, c_s an initialization function, and C_t ⊂ C a set of terminal configurations. We then define different instantiations of S for the different (morphological, syntactic, morphosyntactic) parsing tasks. In each instantiation, a transition sequence y for x is a sequence of configurations that are obtained by applying transitions t₁ …t_n ∈ T sequentially. That is, starting with an initial configuration c₀ = c_s(x), we find y = c₀, …, c_n such that c_{i +1} = t_{i +1}(c_i) and c_n ∈ C_t. Thus, each y depicts a sequence of decisions that constructs a valid analysis for x at the relevant linguistic level.

To compute Score(y), y is mapped to a global feature vector Φ(y) of size d multiplied by a weights vector $ω→$ of the same size. The global feature vector Φ(y) consists of local feature vectors, each of which is defined via a set of functions ${φi:C→N}i=1d$ which count the occurrences of a prespecified pattern in a given configuration in y. Following Zhang and Clark (2011), we learn the weights vector $ω→∈Rd$ via the generalized perceptron using the early-update averaged variant of Collins and Roark (2004).

Decoding is based on the beam search algorithm, where a number of high-scoring candidate sequences are maintained in the beam in order to mitigate irrecoverable prediction errors that characterize greedy search procedures. At each step, the transition system applies all transitions to all candidates, and keeps the B highest-scoring candidates. During learning, the perceptron algorithm iterates through a gold-annotated corpus. Each sentence is parsed (decoded) with the last known weights, and if the parsed result differs from the gold, the weights are updated. The learning is stopped when overfitting begins.

Our departure point for morphological disambiguation (MD) is the transition system of More and Tsarfaty (2016), currently established as the state of the art for Hebrew MA&D.⁴ The input to the system is a lattice L that captures the range of valid morphological analyses for the input tokens x = x₁, …, x_k, as illustrated in the middle of Figure 1. The goal of the MD system is to select a sequence of contiguous arcs in L which represents the morphological disambiguation of x in context.

Formally, we define for each token x_i its token-latticeL_i = MA(x_i) where each lattice-arc in L_i corresponds to a potential node in the dependency tree. Each lattice-arc has a morphosyntactic representation (MSR) which we define as a tuple m = (b, e, f, t, g) with b and e the beginning and end indices in L, f a form, t a part-of-speech tag, and g a set of attribute:value grammatical properties. L = MA(x) is the sentence lattice obtained by concatenating the token-lattices top to bottom L = MA(x₁) ∘ … ∘ MA(x_k). Now, L represents the full range of valid morphological analyses applicable to x.⁵

The terminal configuration set is defined to be C_t = {(L,top(L),|x|,M)} where M = {m₁,m₂,....,m_l} holds the fully disambiguated path of MSRs (selected arcs) through L.

In order to find this path in a data-driven fashion, we define a parametric model that scores all transitions that can be applied at each step. We define the properties f (form), t (pos tag), g (morphological attribute:value pairs), path (the path in the previously disambiguated token-lattices), and morphs (the set of outgoing morphemes of the current node) and we use unigram, bigram, and trigram combinations of these properties as features for the learning model.⁶ Our beam search decoder then applies at each point in the lattice all possible transitions and selects the B-top scoring candidates at this point. Those that don’t make the B mark, fall off the beam.

Importantly, |M|, the number of lattice arcs in the path at each stage, is unknown in advance, since different disambiguation decisions between token boundaries may end up with different path lengths. This can be seen in the lattice of Figure 1, where path lengths vary between 4–7 arcs. This is a thorny issue, because it violates a basic assumption of beam search decoding—that the number of transitions is a deterministic function of the input and is known in advance. Such length discrepancies may lead to preferring short sequences in the beam due to reaching the end goal early, or preferring long sequences, due to artificial inflation of scores based on the multitude of features.

Given a sequence of selected lattice arcs for the input sequence x, we can define the syntactic dependency representation for x as a dependency tree where each lattice arc corresponds to a node in the dependency tree. Let R be a set of dependency types and let M = m₁…m_l be the sequence of l arcs selected by the MD component.⁸ We denote a dependency graph for the sequence M = m₁…m_l as G_M = (V_M,A_M), where V_M is a set of nodes corresponding to the arcs of M and A_M ⊆ V_M × R × V_M is a set of labeled arcs between the elements of V_M.

A configuration represents a partial analysis of the input sentence, where the morphemes on the stack σ are partially processed morphemes, the morphemes in the buffer β are those waiting to be processed, and the arc set A represents a partially built dependency tree (Kübler et al., 2009, Chapter 3). Unless specified otherwise, the set of terminal configurations is C_t = {(σ, β, A)} where β = [] and |σ| = 1.⁹

There are various options for defining transitions over such configurations in the dependency parsing literature for English. In particular, three transition systems have been successfully applied to English as well as other languages (cf. Ballesteros and Nivre [2016]):

• Arc Standard: A straightforward method of bottom-up left-to-right incremental parsing as proposed in Nivre (2004). We assume the definition by Kübler et al. (2009).

• Arc Eager: Following Abney and Johnson (1991), Arc Eager defines a variant of Arc Standard that allows to eagerly attach a right-dependent to its head while allowing more dependents to attach to it. We assume the definition by Kübler et al. (2009).

• Arc (Z)Eager: In our reproduction of the state-of-the-art results presented by Zhang and Nivre (2011) for English, we discovered in the code a variant of Arc Eager that we call Arc (Z)Eager, which has interesting subtle variations from Arc Eager, including a second stack holding head nodes, and certain hard constraints on the application of several transitions.¹⁰

An empirical study by Nivre (2008) compares the performance of Arc Standard and Arc Eager for 13 languages, amongst them Arabic and Turkish, both considered MRLs with some degree of word-order freedom. For these languages, Arc Standard slightly outperformed Arc Eager. On a different but related note, our preliminary experiments on English and Hebrew show that the Arc ZEager variant always outperforms Arc Eager. However, the question which of the two, Arc-Standard or Arc-ZEager, will be more suited for parsing Hebrew, remains open for our empirical investigation in Section 3.

A significant contribution of Zhang and Nivre (2011) is their proposal of a set of rich non-local features (RNF) for Arc ZEager, adding higher-order information previously found only in graph-based parsers. To facilitate a fair comparison of Arc Standard to Arc (Z)Eager, we have to adapt the feature set of Zhang and Nivre (2011) to the different arc system (to the extent that this is possible), and to the different language type. In particular, the RNF set depends on word order, by encoding the arc direction explicitly. We address the order-dependence of RNF by defining a parallel set of features that is suitable for the more flexible word order in MRLs, and that is applicable to Arc-Standard. We call this feature set rich linguistic features (RLF). The essence of the two feature sets is the same, but we replace features relying on positions of nodes with features relying on the labeled grammatical functions of these nodes.¹¹

To construct our features, we define properties that capture the linguistic information of selectional preferences and subcategorization frames (Tesnière, 1959; Chomsky, 1965). To capture the distributional characterization of subcat frames, we define sf_p to be a multiset of part-of-speech tags of the dependents of a given head. To capture the functional characterization of subcat frames, we define the sf_f referring to the multiset of function labels of all dependents of a given head. For valency, we define the properties v_sf, referring to the number of dependents of a given head. For capturing selectional preferences in flexible word order environments, we define order-agnostic bilexical labeled-dependency features, generated separately for each dependent.

Finally, we augment syntactic features with morphological properties. Our augmentation operator allows for creating multiple instances of the same feature, with and without morphological properties.

Given our morphological and syntactic components, we seek an integration such that morphological information aids syntactic disambiguation and vice versa.

We propose to literally embed the two standalone configurations into a single configuration, and to apply transitions via a coherent logic we call a strategy that chooses which processor to apply at a given state.

We initialize the embedded MD configuration c_md with the MD transition system initialization function, as defined in Section 2.2, but leave c_dep empty, with σ = β = [] empty stack and buffer. Also, as before in c_dep, A = ∅. A configuration c_j is terminal if and only if c_md and c_dep are both terminal configurations of their respective systems.

Our baseline morphosyntactic parsing strategy is simply a pipeline that first applies the morphological component which selects the best output morpheme sequence, and then applies the dependency parser to it.

We aim to test the hypothesis that joint syntactic and morphological disambiguation is better than a pipeline by empirically comparing the Pipeline, MDFirst and ArcGreedy₃ parsing strategies in our unified transition-based morphosyntactic framework.¹²

We use the Modern Hebrew section of the SPMRL shared task (Seddah et al., 2014), derived from the Hebrew Unified-SD version of Tsarfaty (2013). For the purpose of this work, we harmonized the treebank annotation scheme with the annotation scheme of the lexical resources of Itai and Wintner (2008), and in particular the HEBLEX lexicon of Adler and Elhadad (2006). We use the standard train/dev/test sets split, train on the train set (5,000 sentences) with a detailed investigation on dev (500), and confirm our results on test (716).

We implemented from scratch a fully integrated, transition-based, multilingual natural language processor, written in Go.¹³ Our implementation uses a general purpose morphological analyzer, which for Hebrew is backed by the BGU HEBLEX lexicon (Adler and Elhadad, 2006). We implemented the morphological disambiguator, dependency parser, and joint integration strategies defined herein. We implemented and experimented with both the Arc Standard and Arc ZEager transition systems.¹⁴

In MRLs, out-of-vocabulary (OOV) tokens pose a great challenge to parsing. A raw token may have not been observed during training, even though all its morphemes have been observed in other contexts. To gauge the effect of such OOV items on the quality of Hebrew parses, we evaluate the system in two different scenarios. In the first, infused scenario, we verify that each lattice contains the gold morphological analysis. That is, if the gold path is not present in L = MA(x) (hence, an OOV), we automatically infuse the gold path into L. We contrast this with uninfused scenarios, where we use a realistic morphological analyzer with its (incomplete) lexical coverage as is, compliant with Adler and Elhadad (2006).

In all experiments, we used a beam of size 64, which, in our preliminary experiments on dev, gave better results for the joint models than a beam of 32, and in any event no worse results than a beam of 128. To avoid both overfitting and underfitting, we define a stopping condition for the training procedure, which we test in each training iteration. During training, we use a sliding window of three iterations and select the first model that precedes two sequential scores-drop on dev.

For pipeline models, we test distinct stopping conditions for the morphological and the syntactic models, each based on its own standalone scores. For joint models, we test the stopping condition with respect to a single overall dependency F₁ score, which we define shortly.

Evaluating joint morpho-syntactic dependency parsing performance is non-trivial, because the gold and parse trees may have a different number of nodes, which precludes the application of standard attachment scores; it suffices that an incorrect segmentation occurs early in the sequence, then off-by-one indices in the remainder of the sentence deem the rest of the arcs incorrect (Tsarfaty et al., 2012).

Let us illustrate this effect. Consider the Hebrew phrase “bbit” (translated “in the house”) that appears as a single space-delimited token. Now consider the two following MD alternatives, with and without the Hebrew covert definite article. We also include here the indices of the disambiguated morphemes in their linear order:

• Gold MD: 1.b(“in”) 2.h(“the”) 3.bit(“house”)

• Predicted MD: 1.b(“in”) 2.bit(“house”).

Further assume that both the Gold and Predicted dependency trees contain the correct dependency arc between b (“in”) and bit (“house’’) labeled pobj. In simple LAS terms, the arcs that would be compared for the purpose of evaluation are:

• Gold Dep: pobj(1,3), det(3,2)

• Predicted Dep: pobj(1,2).

So the pobj predicted arc will be considered an error, even though the relation between forms is correct, and accordingly both UAS and LAS will be 0.

Now, the parser will be credited for identifying the pobj arc correctly, as desired, and the dependency scores will be: Pr = 1, Re = 0.5, and F₁ = 0.67.

Tables 1–4 present our morpho-syntactic parsing results for each of our different systems in all, pipeline and joint, strategies. We report F₁ scores, both MD Full and MD POS for morphological disambiguation (MD), and both unlabeled and labeled F₁ scores for the dependency trees (Dep). Tables 1 and 3 present results on the Modern Hebrew dev set, and Tables 2 and 4 confirm our results on the test set.

Table 1 presents parsing results for infused morphological lattices; that is, ambiguous MA lattices that are guaranteed to also include the correct MD path in them. In these experiments, we see that MD results in joint parsing strategies (MDFirst, ArcGreedy) always improve upon the MD standalone/pipeline results. In particular, all MD results across the joint strategies are very close. We observe only a minor advantage for Arc-Zeager over Arc-Standard for both joint strategies. This increase in MD accuracy unfortunately comes at the expense of syntax, where we observe a slight drop (up to 0.5 point in [un]labeled F₁) when switching from pipeline to joint strategies.

We confirm this trend on the test set in Table 2, where we use the same models in the infused settings to parse the standard test set. For MD, all joint results are better than the respective pipelines (although now Arc-Standard slightly improves upon Arc-Zeager in the ArcGreedy strategy), while dependency parsing results drop in joint scenarios (a slightly larger drop than on dev).

Tables 3 and 4 present parsing results for the more interesting scenario, a realistic parsing scenario where we use uninfused lattices—ambigous lattices obtained by an existing broad-coverage morphological analyzer, which are not (and cannot be) guaranteed to always also include the correct path. As expected, on both the dev set (Table 3) and test set (Table 4), the results drop relative to the respective infused scenarios (Tables 1 and 2, respectively), as some elements from the correct path and tree are no longer reachable within the search space. At the same time, it is interesting to observe that for both dev and test, all MD scores (Full/POS) as well as dependency scores (un/labeled) are better in joint parsing. The specific differences between the joint strategies and transition systems do not matter very much— the robust empirical trend is that switching from pipeline to joint improves both MD and dependency parsing performance.

It is interesting to inquire why in the infused scenario, on both dev and test, dependency parsing results in the joint strategies drop relative to the respective pipelines. At it turns out, in case the correct analysis of a rare (OOV) token has been injected artificially into the lattice, training on these lattices may turn out to be misleading. Injecting a correct but rare MSR may lead to an artificial “certainty” as to its appropriate syntactic context. Then, if the parser does not apply robust statistics on the general behavior of rare/OOV items in different syntactic contexts (as would be the case in joint uninfused scenarios), selecting the injected MD may lead to a wrong syntactic decision.

The main message coming out of our experiments is that joint morphological disambiguation and syntactic parsing in this transition-based framework is preferred to pipeline settings, in line with the hypothesis that syntactic information aids morphological disambiguation. Furthermore, it is reassuring to observe that when parsing uninfused lattices, as in the more realistic scenario, dependency parsing results improve upon pipeline scenarios, corroborating the findings of Seeker and Centinoglu (2015) in graph-based frameworks and of Cohen and Smith (2007) and Goldberg and Tsarfaty (2008) in phrase–structure parsing.

To put our end-to-end system performance in context, Tables 5 and 6 present our best results for dependency parsing in a pipeline architecture, assuming gold morphology, on the dev set and the test set, respectively. We compare these results with studies that parsed the same data sets. As Table 5 shows, our parser significantly outperforms the state-of-the-art parser by Goldberg and Elhadad (2011), so far considered the de facto standard for Hebrew parsing.¹⁶ As shown in Table 6, the parser also outperforms the results reported by most (though not all) SPMRL shared tasks participants, using the same data and same split.

Such gold morphology settings are of course not suited for realistic parsing scenarios. So, in Table 7 we compare our best end-to-end parsing results to the most recent dependency parsing results in realistic scenarios on the same data (by (Seeker and Centinoglu 2015). Here our best pipeline and joint systems outperform the previously reported pipeline and joint results, thus presenting a new state of the art for Hebrew dependency parsing. Moreover, these results are obtained within a unified formal framework in a single “all-included” implementation, providing a further practical advantage of not having to maintain and train separate standalone components.¹⁷

To shed more light on the particular ways in which the joint system improves performance over the pipeline, we conducted a qualitative error analysis in 100 sentences from the Modern Hebrew standard dev set, when parsed in the more realistic uninfused scenario. More concretely, we sampled 100 sentences from our parsed corpus and a linguist manually assigned each error to one of 10 linguistic categories. We then clustered the categories into four different types.

• • 

  TYPE 1 errors include true semantic ambiguity, where additional semantic and world knowledge is required for disambiguation.

• • 

  TYPE 2 errors include categories that transcend different levels of linguistic structure, for example, when morphological segmentation errors affect syntactic disambiguation.

• • 

  TYPE 3 errors include parsing errors that stem from idiosyncrasies of the data and peculiarities of the SPMRL annotation scheme,

• • 

  TYPE 4 (other) errors include parse errors that pertain to linguistic structures that characterize Semitic phenomena.

Table 8 shows, for each error category, the number (and percentage) of occurrences of that error in the pipeline versus joint settings. The most outcome is that the type that shows the largest decrease in joint scenarios relative to pipeline scenarios belongs to TYPE 2, reflecting phenomena directly related to the morpho-syntactic interface. Moreover, we also see a decrease in the errors concerned with the lexico-syntactic interface (e.g., solving PP attachment ambiguity), which turn out to also benefit from the joint settings. With the other types of errors, there is no clear advantage for joint parsing, and we would not expect one. TYPE 3 errors have to do with train-set inconsistencies, under-specification, or errors in the gold trees. TYPE 4 errors stem from linguistic phenomena which appear harder to disambiguate, and they are equally difficult across scenarios.

Monolingual MA&D for Modern Hebrew has been previously addressed in standalone settings using Hidden Markov Models (Bar-haim et al., 2008; Adler, 2007). While these results are adequate for some downstream applications, using Adler’s MA&D for dependency parsing, for instance, significantly harms parsing performance (Goldberg and Elhadad, 2010). More recently, More and Tsarfaty (2016) presented a standalone transition-based MA&D which jointly solves morphological segmentation, tagging, and feature assignment, presenting new state-of-the-art Hebrew MA&D, providing the starting point for our study.

In terms of end-to-end dependency parsing for Hebrew, Goldberg and Elhadad (2010) were the first to evaluate the impact of predicted morphology compared to gold morphology across different (transition-based, graph-based, easy-first) frameworks. They demonstrated a significant loss in accuracy for all models in predicted morphology settings, and concluded with a suggestion to attempt joint processing. Recently, Straka et al. (2016) presented UDPipe, a toolkit with standalone components for morphological analysis, segmentation, tagging, features assignment, and dependency parsing—again using a pipeline architecture, with no way of interleaving the different decisions, as we strive to do here. This work aims to cover all stages of UDPipe, but within a joint architecture, allowing the use of information from any layer when disambiguating another.

Joint morphological and syntactic processing has been addressed in the context of phrase-structure parsing for Semitic languages, showing empirical advantages over pipeline architectures (Goldberg and Tsarfaty, 2008; Cohen and Smith, 2007; Green and Manning, 2010). In the context of dependency parsing, Bohnet and Nivre (2012) and Bohnet et al. (2013) integrated tagging and dependency parsing, improving state-of-the-art accuracy for a set of typologically different languages. Andor et al. (2016) use the joint transition system proposed by Bohnet and Nivre (2012), and improve it using a globally normalized neural network. These systems address joint morpho-syntactic analysis for disambiguated words, but without addressing the issue of segmenting and disambiguating raw input tokens.

Seeker and Centinoglu (2015) explore the idea of joint morphological and syntactic parsing, including morphological segmentation, in a graph-based framework. Their system integrates two standalone components that reach agreement via a dual-decomposition setup. However, they report suboptimal performance on the standard Hebrew benchmark. For various Chinese parsing tasks, joint systems for word segmentation and syntactic parsing have been shown to outperform pipeline settings (Li et al., 2011; Zhang et al., 2014), but these systems assume transitions over equal-length character-based sequences, and thus they are not applicable to the setup of variable-length lattice paths, as demonstrated in Figure 1.

With the surge of interest in deep learning for NLP (Goldberg, 2016), research in dependency parsing seeks to replace engineered feature models with neural networks that induce a model automatically (Chen and Manning, 2014; Zhou et al., 2015; Andor et al., 2016). Furthermore, the concept of word embedding introduced by Mikolov et al. (2013) allows for words to have vector representations, such that syntactic and semantic similarities are embodied in the vector space. However, these kinds of architectures are not immediately applicable to parsing Hebrew and other MRLs. Pretraining word embeddings is non-trivial for ambiguous input tokens, unless resorting to pipeline “segmentation-first” scenarios. Similarly, parsing architectures based in RNNs require morphologically disambiguated forms as input, which prevents syntax from improving morphological disambiguation, as we argue for here.

In the future, we intend to augment the architecture we present here with neural network models for both the morphological and syntactic models, in a way that would allow them to effectively interact and affect one another, in the hope to lead towards further improvements in both tasks.

We present a novel joint transition-based framework for morpho-syntactic parsing, designed to solve end-to-end dependency parsing in realistic scenarios. We consider the properties of MRLs and directly address the disambiguation of raw input tokens exploiting larger syntactic contexts. We apply this system to Modern Hebrew, and our empirical results support the long-standing conjecture that MA&D can greatly benefit from syntactic parsing. We present a new set of state-of-the-art Hebrew parsing results, in both pipeline and joint scenarios, which then serve as a strong baseline for exploring future neural joint morpho-syntactic architectures that would potentially improve performance on both tasks.

We thank Joakim Nivre, Yue Zhang, and Yoav Goldberg for comments and suggestions, and four anonymous reviewers for their comments on earlier drafts. We further thank Shoval Sadde, Yochay Gurman, and Dan Bareket from the ONLP Lab at the Open University of Israel for critical discussion of the data and the empirical results. This research was supported by a Starting Research Grant from the European Research Council (ERC-StG-677352), and a grant from the Israel Science Foundation (ISF-1739/26), for which we are grateful.