Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations

The task of formal semantic parsing is to map natural language expressions (words, sentences, and texts) to unambiguous interpretable formal meaning representations. The components of these meaning representations can be divided into two parts: the domain-independent logical symbols (such as negation $¬$⁠, conjunction ∧, disjunction ∨, equality =, and the quantifiers ∃, ∀), and the non-logical symbols (the concepts and relations between them—the predicates), possibly tailored to a specific domain. It is the latter component that forms the focus of this article, as we see several shortcomings in the way predicates are represented in mainstream computational semantics, since this is usually done by combining a lemma, part-of-speech tag, and sense number. This representation of concepts is not only language-specific, but also doesn’t exploit the power of pre-trained language models used in neural approaches, currently the state of the art in semantic parsing (Bai, Chen, and Zhang 2022; Martínez Lorenzo, Maru, and Navigli 2022; Wang et al. 2023).

We illustrate this issue with an example. Figure 1 shows a meaning representation in the form of a directed acyclic graph, as is common in currently used frameworks in computational semantics (Allen, Swift, and de Beaumont 2008; Banarescu et al. 2013; Oepen et al. 2020a; Abzianidze, Bos, and Oepen 2020; Bos 2023). The nodes represent concepts that are symbolized by the lemmas of the content words (nouns, verbs, adjectives, adverbs) that triggered them. As content words arse often polysemous, they are usually disambiguated using one of the standard lexical ontologies, including WordNet (Fellbaum 1998), OntoNotes (Hovy et al. 2006), VerbNet (Bonial et al. 2011), and BabelNet (Navigli and Ponzetto 2012). This can be done by adding a part-of-speech sign and sense number suffix to the symbol, and is mainstream practice across a wide spectrum of semantic formalisms, such as Abstract Meaning Representation (AMR) (Banarescu et al. 2013), BabelNet Meaning Representation (BMR) (Martínez Lorenzo, Maru, and Navigli 2022), Discourse Representation Structure (DRS) (Bos et al. 2017), Prague Tectogrammatical Graphs (PTG) (Zeman and Hajič et al. 2020), and Elementary Dependency Structure (EDS) (Oepen and Lønning 2006). However, the (interrelated) problems that we observe with this approach are the following:

• The representations of concepts are usually not normalized: They are represented in various ways based on synonym lemmas of the same language or translated lemmas for other languages;

• The predicate symbols possess minimal or no inherent semantic structure. This makes it impossible to determine how they are related to other predicates without access to external knowledge bases;

• The predicate symbols are essentially atomic. Yet, a neural network’s tokenizer will typically break down the symbols into meaningless sequences of characters to reduce the size of its vocabulary.

The first problem can be illustrated with a simple example. Consider the English synonyms car and automobile. In most current approaches, their concepts would be represented with different predicate symbols (e.g., car.n.01 and automobile.n.01, following WordNet), even though they share the same lexical meaning. For non-English lexicalizations for this concept, for instance Italian macchina or French voiture, one could opt to use English-based predicate symbols (as is done by Abzianidze et al. 2017) or use a multi-lingual lexical ontology (Martínez Lorenzo, Maru, and Navigli 2022), but these solutions don’t apply any kind of normalization: The same meaning is represented by different predicate symbols.

The second problem has risen more to the foreground with the rise of distributional semantics to capture word meanings. While certain predicate symbols exhibit an accidental internal structure that provides additional insight about them (e.g., blackbird.n.01 is a kind of bird.n.01), for most symbols, it is not possible to determine, without external resources, that dog.n.01 is closely related to and compatible with animal.n.01 or puppy.n.01. This is in stark contrast with semantics based on embeddings where meanings are represented by large vectors based on their contextual occurrences in corpora (Mikolov, Yih, and Zweig 2013). Some attempts have already been made to close the gap between semantic networks and semantic spaces (Rothe and Schütze 2015; Saedi et al. 2018; Scarlini, Pasini, and Navigli 2020a; Lees et al. 2020).

The third problem is perhaps more of a theoretical issue, with the desire to use a clean and sound methodology for composing meanings. From an engineering point of view, what happens under the hood of a semantic parser isn’t that important as long as it produces satisfactory accuracy scores. However, a clear disadvantage is that predicted concepts are not guaranteed to exist in the external knowledge base (e.g., WordNet).

These three problems were not particularly pressing in the past or considered problematic, perhaps mainly because word sense disambiguation was seen as a separate task in the sequential symbolic pipeline of traditional semantic parsing. However, the advent of neural methods in semantic processing has magnified these concerns. Neural networks, due to their statistical nature, are confined to their training data distribution, and are therefore struggling in understanding and generating out-of-distribution concepts (Johnson et al. 2017; Lake and Baroni 2017; Kim and Linzen 2020; Li et al. 2023; Groschwitz et al. 2023; Zhang et al. 2024). Additionally, neural networks tend to find typographical correspondences between the input words and output concepts (Edman et al. 2024), thereby copying from the input which, we argue, artificially inflates parsing performance scores. Nevertheless, we believe that pre-trained language models possess the capability to comprehend the semantics of unseen words and concepts within context, but the current representations used in semantic parsing form an obstacle to exploit it.

The following scenario illustrates this issue. Consider a neural semantic parser trained on a corpus that pairs English sentences with meanings that encode concepts in the standard symbolic format based on a lemma and a sense number. We give this parser the sentence in Figure 1 about birdwatcher John who spotted a hobby, a rare type of falcon. Assume that the word hobby is not in the training data at all. It is very likely that this kind of parser would associate the word hobby with the incorrect concept hobby.n.01, the “spare-time activity” sense, because it learned how to lemmatize from other examples and the suffix “n.01” appears to be the most frequent noun sense. By chance, the parser could produce the correct concept hobby.n.03 (the bird sense), maybe because the suffix “n.03” is often associated with lemmas that end in “by.” In our view, we don’t want to develop semantic parsers that make a wild guess for unknown concepts. Instead, we would like to develop semantic parsers that make an educated guess: perhaps it could come up with a concept that it has seen during training, for example, bird.n.01 or even falcon.n.01 that appears in a similar context as the sentence it tries to parse. In this article, we introduce and evaluate a method for integrating pre-trained language models with symbolic meaning representations that utilize taxonomical encodings.

In the approach we follow, we radically deviate from the view of representing non-logical symbols based on lemmas, part-of-speech tags, and sense numbers. Instead we introduce a way to encode concepts and relations with internally structured blocks of meaning based on external lexical ontologies or taxonomies. These predicate symbols are not directly interpretable for human beings as they do not correspond to units of language. For instance, the taxonomical encodings for cat and dog share a common prefix that tells us that both are of the category mammal. The encodings for adjectives like good and bad will be very similar, with the only difference indicating that they are antonyms. The idea, illustrated in Figure 2, draws inspiration from methods in image classification (Mukherjee, Garg, and Roy 2021; Bertinetto et al. 2020).

Our research questions are the following. First of all, how can we take existing lexical ontologies as a starting point to encode symbols for concepts (nouns, verbs, adjectives, adverbs) in a compositional way? Second, are current neural models actually capable of learning these abstract symbols that have no direct connection with any surface form of language, without reducing parsing performance too much? Furthermore, how can we intuitively assess the model’s understanding of the hierarchy of concepts, and does the model trained on taxonomical encodings demonstrate a superior performance? Finally, do taxonomical encodings of concepts give the semantic parser the ability to make sense of out-of-distribution concepts by assigning a meaning that is compatible with or close to the ground truth, rather than producing a symbol based on the lemma of the word and hoping for the best? In other words, can the parser make an educated guess based on distributional meaning rather than on superficial features when it encounters word meanings that it has never seen before, taking advantage of pre-trained language models?

Our work’s primary contribution lies in presenting a taxonomy-based approach to meaning representation, which is very distinct from prior methods. Accordingly, we connect symbolic and neural techniques in a semantic parsing system. It is symbolic because the semantic parser generates interpretable and transparent meaning representations. It is neural because it harnesses the power of distributional meaning models and neural networks to enhance robustness in unexpected circumstances. A direct consequence of our new approach to semantic parsing is that current evaluation methods, which are based on graph matching, are not sufficient for our purposes, as comparing concepts will be beyond simple string matching and needs to be based and implemented using semantic rather than syntactic similarity. Hence, to assess the capabilities to deal with out-of-vocabulary concepts, we have developed a new challenge test and new semantic evaluation tools. Furthermore, we probe the embeddings of concepts within neural models to evaluate the encoded taxonomical information. Our experiments reveal that our new representation-based parser performs comparably to the original representation in standard tests but excels in predicting unknown concepts on challenge sets. Additionally, the probing tests indicate that neural models trained with the new representation learn more taxonomical information compared to the original representation.

Early approaches to semantic parsing were mostly based on rule-based systems with compositional semantics defined on top of a syntactic structure obtained by a parser (Woods 1973; Hendrix et al. 1977; Templeton and Burger 1983). The development of syntactic treebanks contributed to the creation of robust statistical syntactic parsers, thereby facilitating semantic parsing with broad coverage (Bos et al. 2004). The emergence of neural methodologies and the availability of extensive semantically annotated datasets (Banarescu et al. 2013; Bos et al. 2017; Abzianidze et al. 2017) marked a shift in semantic parsing techniques, diminishing the emphasis on syntactic analysis (Barzdins and Gosko 2016; van Noord and Bos 2017; Bevilacqua, Blloshmi, and Navigli 2021). The introduction of pre-trained language models within the sequence-to-sequence framework led to further enhancements in parsing accuracy (Samuel and Straka 2020; Shou and Lin 2021; Lee et al. 2021; Bevilacqua, Blloshmi, and Navigli 2021; Lee et al. 2022; Bai, Chen, and Zhang 2022; Martínez Lorenzo, Maru, and Navigli 2022; Wang et al. 2023).

Various meaning representations have been the target for semantic parsing—for excellent recent overviews, see Oepen et al. (2020b) and Sadeddine, Opitz, and Suchanek (2024). Dominating the field of semantic parsing is AMR, facilitated by the large supply of annotated data and the simplicity of its meaning structures, that are based on simple directed acyclic graphs. There are various extensions proposed to enrich AMR. These are, among others, BMR, which incorporates multi-lingual semantic resources (Martínez Lorenzo, Maru, and Navigli 2022), and Uniform Meaning Representation (UMR), which includes discourse level phenomena (Gysel et al. 2021). In this work we focus on DRS, a richer meaning representation, a semantic formalism that drew substantial interest in computational semantics (Bos et al. 2004; Evang and Bos 2016; van Noord et al. 2018b; van Noord, Toral, and Bos 2019; Evang 2019; Liu, Cohen, and Lapata 2019; van Noord, Toral, and Bos 2020; Wang et al. 2021; Poelman, van Noord, and Bos 2022; Wang et al. 2023; Zhang et al. 2024).

A common way of evaluating semantic parsing is to compare parser output with a gold standard using graph overlap (Allen, Swift, and de Beaumont 2008; Cai and Knight 2013; van Noord et al. 2018a). A well-known implementation is Smatch (semantic match), which measures the structural similarity by mapping graphs into triples and determining the maximum amount of triples that are shared by two semantic graphs, computed by taking the harmonic mean of precision and recall of matching triples (Cai and Knight 2013). Smatch is mostly used to assess AMR parsing, but not exclusively, and Poelman, van Noord, and Bos (2022) adopt it for DRS parsing. The original Smatch implements matching of semantic material without further nuances, so several variations and extensions of Smatch have been developed to enhance semantic evaluation (Damonte, Cohen, and Satta 2017; Cai and Lam 2019; Opitz, Parcalabescu, and Frank 2020; Wein and Schneider 2022; Opitz 2023).

Cai and Lam (2019) argue that more weight should be given to triples that form the core of the meaning expressed by a semantic graph. They propose a variant of Smatch that takes root distance into account by reducing the significance of triple matches that are further away from the root of the graph (Cai and Lam 2019). In the meaning representations of our choice, DRS, there is no designated root, so we cannot adopt this variant of Smatch straightaway. Although we think the idea of giving more importance to triples that form the core of the meaning expressed by a text is interesting, we believe more research is required to establish what exactly constitutes this and we therefore will consider this outside the scope of this article.

Opitz, Parcalabescu, and Frank (2020) argue that the “hard” matching of Smatch is not always justified and propose S²Match (soft similarity match), by implementing a graded semantic match of concepts with the help of a distance function that computes a number between 0 and 1. The distance function can be anything that fulfills its purpose. Opitz, Parcalabescu, and Frank (2020) showcase their idea using GloVe embeddings to obtain a similarity score between lemmas. Wein and Schneider (2022) propose LaBSE embeddings (Feng et al. 2022) to compute the similarity between two concepts in different languages for cross-lingual comparison of meaning representations. However, these choices of distance functions ignore scenarios where different concepts are expressed by the same lemma. We embrace the use of S²Match in our work but will replace the distance function by an operation that measures the ontological distance between two concepts or relations.

We make heavy use of the lexical ontology WordNet, a handcrafted electronic dictionary (Fellbaum 1998). In WordNet, words are organized around synsets, that is, sets of words that have similar meanings. A synset consists of one or more words; an ambiguous word (a word with more than one sense) is placed into several synsets, one for each distinct meaning. Synsets are connected to each other by several semantic relations (see below). Each synset has a unique identifier, a 9-digit number based on the byte offset in the WordNet database, where the first number identifies the part of speech.

The original WordNet was designed for English (Fellbaum 1998). Subsequent efforts have been undertaken to establish WordNets for various languages and to develop multilingual lexical resources (Navigli and Ponzetto 2012; Vossen 1998; Bond and Foster 2013), or to include WordNet into a formal ontology (Gangemi et al. 2003) and to integrate it with Wikipedia (Suchanek, Kasneci, and Weikum 2008; Speer, Chin, and Havasi 2017).

Only parts of speech of content words are included in WordNet: nouns (n), verbs (v), adjectives (a), and adverbs (r). There are several ways to refer to a specific synset: by using the unique identifier, or, more commonly, by a combination of lemma, part of speech, and sense number of one of its members. For instance, the English noun hobby is placed into three synsets in Princeton’s WordNet for English: the first sense, hobby.n.01, is glossed as “an auxiliary activity,” and two of its other synset members are pursuit.n.03 and avocation.n.01; hobby.n.02 denotes the sense of plaything for children, and hobby.n.03 refers to the bird of prey with the scientific name Falco subbuteo.

The power of WordNet manifests itself by the relations between synsets that it offers. The hypernym and hyponym relations connect generic with specific synsets. For instance, the direct hypernym of hobby.n.03 is falcon.n.01. Synsets that share the same hypernym are called co-hyponyms. For example, hobby.n.03 and peregrine.n.01 are co-hyponyms, as they are both falcons. Verbs are similarly organized in WordNet, but the term troponym is used instead of hyponym to indicate a more fine-grained sense of a verb. For instance, falcon.v.01 is a troponym of hunt.v.01. Some verbs are related to other verbs via the entailment relation, for example, oversleep.v.01 entails sleep.v.01, and some verbs have derivationally related forms corresponding to nouns, for example, hunt.v.01 is related to hunt.n.08.

Adjectives and adverbs are quite differently inserted into WordNet than nouns and verbs are. Adjectives are arranged by falling into one of the categories of head and satellite, the former playing a more pivotal role, and the latter a specialization of a certain adjective. For some adjectives, there exists the antonym relation between synsets of head adjectives, indicating opposite meanings. For instance, good.a.01 and bad.a.01 are antonyms, with satellites cracking.a.01, superb.a.02, among others, for the former, and awful.a.01 and terrible.a.02 (among many others) for the latter. Some adjectives are connected to attribute nouns, for example, fast.a.01 has the attribute synset speed.n.02. Adjectives are also connected to their derivationally related forms of verbs and nouns. Some adverbs are connected to adjective by the pertainym relation, for example, quickly.r.01 is a pertainym of quick.a.01. However, there are several adverbs that have no connection with other synsets in WordNet.

For instance, the WPS of the first and third (semantically unrelated) senses of the noun hobby is low: WPS(hobby.n.01, hobby.n.03) = 0.087, whereas the similarity between hobby (the bird) and falcon is high: WPS(falcon.n.01, hobby.n.03) = 0.963. Note that the similarity metrics mentioned are grounded in WordNet’s taxonomy and typically are most appropriate for noun comparisons, and less so for other parts of speech. In Section 4, we delve into the distinct taxonomies of verbs, adjectives, and adverbs in WordNet. These structural differences lead to inaccuracies in measuring concept similarity for non-noun categories using the existing WordNet taxonomy. To address this issue, we calculate the similarity using the taxonomy encodings described in Section 4 instead of using the original WordNet taxonomy.

One could also view our taxonomical encoding as a vector, and apply cosine similarity for assessing the similarity between two concepts. However, we won’t get the nuances that we need if we follow this approach. This is because cosine similarity would not take the position of the element within the vector into account, thereby overlooking the inherent hierarchical structure of our taxonomical encodings. For instance, considering Figure 2, the cosine similarity for “falcon” and “chaffinch” would be the same as that for “falcon” and “dog,” and this is not what we would expect, as the former pair of concepts is more similar than the latter.

Symbolic meaning representations consist of the logical and non-logical parts. The non-logical symbols, the predicates and relations, define the concepts in the domain of interest. Here we assume an open-domain approach where nouns, verbs, adjectives, and adverbs are mapped to predicates taken from an ontology, proper names¹ and numbers are mapped to literals, and prepositions and implicit arguments are mapped to an inventory of roles and relations.

In NLP, distinct formats for non-logical symbols (also known as predicate symbols) have been adopted, ranging from words, lemmas, a combination of a lemma and a sense number, to entries in a lexical ontology. In AMR (Kingsbury and Palmer 2002), predicate symbols are only partially disambiguated. Some symbols are derived from PropBank framesets and formatted as lemma-sense (e.g., see-01) but most predicates do not include senses and are simply represented by their corresponding lemma. BMR (Martínez Lorenzo, Maru, and Navigli 2022) follows the graph structure of AMR but the predicates are encoded by leveraging the multilingual semantic network of BabelNet, interpreting the non-logical symbols from WordNet, Wikipedia, and other resources. In the DRS representation of the Parallel Meaning Bank (PMB) (Abzianidze et al. 2017), predicates for noun, verbs, adjectives, and adverbs follow the corresponding WordNet synsets and adhere strictly to the lemma-pos-sense format (see Figure 1).

Hence, one important subtask of semantic parsing is Word Sense Disambiguation (WSD), the process of identifying the appropriate meaning of a word within its context. Traditionally, this is approached as a classification task with the goal of selecting the correct sense from a set of predefined sense inventories (Bevilacqua et al. 2021). In contrast, the task of concept prediction in semantic parsing is treating WSD as a generative task. Here, semantic parsers are expected to generate the correct word sense without being given the inventories of senses. This presents a significant challenge, as it is considered currently “impossible” to accurately generate word senses without external knowledge sources, particularly when the word senses have not been encountered in the training data (Groschwitz et al. 2023).

In the case of semantic parsing, the sense number system acts as an instrument for sense prediction. However, numbering senses is fairly arbitrary, only constrained by the tendency that senses encountered frequently in corpora get assigned a low sense number. Consequently, for concepts not encountered in the training data, predicting whether a word corresponds to sense number 3, 4, or 5 holds no distinguishable difference: The best strategy to use for a WSD component would be choosing a low sense number, like 1 or 2. Our new representation for concepts presented in Section 4 addresses this issue by refraining from using sense numbers and instead incorporating taxonomical information in concept representation.

The aim of this article is to develop and evaluate a new way of representing concepts and incorporating them into formal meaning representations of natural language sentences. Another way of representing concepts are sense embeddings, pre-trained vectors extracted from a neural model, usually based on language models trained on a corpus with labeled word senses. Various approaches have been introduced to generate sense embeddings. AutoExtend (Rothe and Schütze 2015) derives synset and lexeme embeddings from word embeddings. Context-AwaRe Embeddings of Senses (Scarlini, Pasini, and Navigli 2020b) uses a semi-supervised approach to producing sense embeddings for the lexical meanings within a lexical knowledge base. SensEmBERT (Scarlini, Pasini, and Navigli 2020a) uses an approach by combing the power of the language modeling and the knowledge contained in a semantic network. Pre-trained sense embeddings are known to improve word sense disambiguation9 (Oele and van Noord 2018; Bevilacqua and Navigli 2020).

However, due to their size and floating point numbers, pre-trained sense embeddings are (obviously) not directly appropriate to be explicitly part of a formal meaning representation such as the one shown in Figure 1. One option would be to compress the embeddings (Andrews 2015) and transform the numbers into integers, but this solution falls outside the objectives of this article. Hence, although we will not explore the use of pre-trained sense embeddings to improve semantic parsing, in Section 6 we will compare them with the sense embeddings extracted from the semantic parsing models that we develop to get an idea how well they reflect the taxonomical hierarchy encoded by WordNet.

In order to run our experiments we need a reasonably sized annotated corpus of sentences and their meaning representations. Several such annotated corpora are available for AMR (Banarescu et al. 2013), and the majority of current semantic parsing methods are developed using AMR datasets. However, for our purposes, we are not able to make use of these linguistic resources because only part of the non-logical symbols (predicates) are disambiguated in AMR, as we outlined in the previous section.²

Instead, we will work with a variant of DRS, the meaning representation proposed in Discourse Representation Theory (Kamp and Reyle 1993). The PMB offers a large corpus of sentences paired with DRSs with concepts represented by WordNet synsets and a neo-Davidsonian event semantics with VerbNet-inspired thematic roles.

The formal language of DRS consists of discourse referents and DRS conditions. DRS conditions are predicates applied to discourse referents, relations between discourse referents or literals, or comparison statements (i.e., equality, approximation, temporal precedence; see Appendix C). DRSs are recursive data structures; complex DRSs can be constructed to express negation, conjunction, and discourse relations.

An example DRS in box format, the equivalent for the meaning representation graph in Figure 1, is shown on the left in Figure 3. However, in our experiments in Section 6, we will use neither of these formats when training our semantic parsing models. Instead, we will use the sequence notation for DRS where variables are replaced by De Bruijnian indices (Bos 2023). The sequence notation is a convenient way for training neural semantic parsers that are based on seq2seq architectures, because there are variables names and a minimal amount of punctuation symbols. Our running example in sequence notation is shown on the right in Figure 3.

There are four parts of speech in WordNet, all with a different ontological organization. Therefore, we describe for each category how we compute its taxonomical encodings. These encodings will be our new way to represent concepts in a formal meaning representation and used to improve semantic parsing. We use Princeton WordNet version 3.0 (Fellbaum 1998), compatible with the PMB.

For the encoding of nouns we will make use of the WordNet hyponym-hypernym relation between synsets. Each noun synset has one or more hypernyms, except entity.n.01, which therefore represents the most general synset. For noun synsets with more than one synset (i.e., indicating multiple inheritance) we consider just one of the possible hypernyms.

This procedure maps all the noun synsets to one large ISA-hierarchy with the top node entity.n.01. Given a synset within this obtained hierarchy, we give each direct hyponym-hypernym edge a label (a single ASCII character, excluding the zero “0”), ensuring that the labels for each co-hyponyms are all distinct.³ Once we have done this for each edge, we can read off a unique sequence of labels for each synset. The maximum length of this sequence is the maximum depth of hyponym-hypernym links in WordNet. We pad encodings with trailing zeroes in order to give each synset encoding the same length.⁴ The number of different labels that we need corresponds to the maximum of co-hyponyms for a synset in WordNet.

Table 1 gives a snapshot of how this labeling works. The resulting taxonomical code gives us a symbolic representation that groups similar concepts (i.e., synsets) together based on their internal structure. The more labels that they share from left-to-right in the encoding, the more they have in common. The more zeroes an encoding has, the more general its synsets is. To distinguish noun synsets from other parts of speech, we attach the prefix “n” to the encoding.

For verb synsets we follow essentially the same procedure as for nouns presented in the previous section making use of the troponym and entailment relations available in WordNet. However, the hierarchy of verbs is much flatter than that of nouns, resulting in too many top nodes (synsets without hypernym). For verb synsets without a hypernym, we create edges to noun synset to which they are derivationally related, as shown in Table 2. For noun synsets inserted in this way, we expand the hierarchy as we did for nouns. To distinguish verb concepts from noun-derived concepts, we attach the prefix “v” to it.

For adjective synsets (Table 3) we create a hierarchical link between satellites and their heads. The head adjectives will be related to noun synsets using derivationally related verb or noun synsets or attribute nouns. To distinguish the adjective encodings, we attach the prefix “a” to it. Antonyms receive the same encodings but are decorated with a positive or negative suffix.⁵ WordNet doesn’t provide information whether an antonym is positive or negative, so we use a simple heuristic to check the prefix of the adjective’s lemma (im-, non-, un-)—see van Son, van Miltenburg, and Morante (2016) and Blanco and Moldovan (2010). Adverbs are linked to adjectives via the pertainym relation and receive the same encoding but with the prefix “r.”

The meaning representations that we use follow a neo-Davidsonian way of representing events (Parsons 1990), where events are related to their participants by a close set of thematic roles, namely, Agent, Theme, Patient, Result, and so on. The inventory of roles is an extension of the hierarchical set proposed in VerbNet (Bonial et al. 2011), extended with roles used in the Parallel Meaning Bank. The elaboration of the complete taxonomy of these roles is outlined in Appendix B. There are also roles to connect non-event entities, for instance those appearing in genitive constructions or noun compounds. Some thematic roles are paired with their inverse roles, for instance, Sub and SubOf. To clearly distinguish between these roles, we use distinct prefixes: “t” denotes a thematic role, whereas “i” signifies an inverse role. We also convert each Discourse Relation and operator into distinct mathematical symbols, as shown in Appendix C. In contrast to roles, these two logical components lack a taxonomy structure, therefore their encoding is straightforward, involving a direct mapping to single-byte symbols.

Now that we have explained how the encoding process works for concepts, roles, operators, and discourse relations, we can put everything together and transform the semantic graph into a graph encoded with the WordNet Identifier and WordNet taxonomical encoding, as illustrated in the graphs in Appendix A. The sequential representations of meanings, which are used for training in Section 6, are illustrated in Table 4.

Several new tools are required to work with the taxonomical encoding of concepts that we proposed in the previous section. First of all, we need to revise the existing way of measuring semantic parsing performance. We will do so by replacing the well-known Smatch metric with one that takes concept similarity into account. Second, we need a new challenge set that measures parsing performance on out-of-distribution concepts. Finally, we need to add an interpretation component to the semantic parser that maps taxonomical encodings back to human-readable concepts.

We adopt the S²Match framework of Opitz, Parcalabescu, and Frank (2020) (see Section 3.2) but replace its distance function to incorporate taxonomical encodings. Recall that Smatch converts a semantic graph into node-edge-node triples and computes a score based on the maximum number of matching triples. In standard Smatch, two triples get a matching score of 1 if and only if there is a perfect match between the two nodes and the edge. S²Match extends this approach by introducing a soft matching between instance triples, where a distance function based GloVe embedding similarity returns a score between 0 and 1. We modify Smatch and S²Match with respect to three issues:

1. We replace the distance function based on word embeddings by the Wu-Palmer Score (see Section 3.3);

2. We not only allow soft matching based on the Wu-Palmer Score for instance triples, but also for role triples;

3. In the implementation of Smatch, triples featuring TOP were discarded since DRS, unlike AMR, does not contain roots.

In the rest of this article we refer to these alterations of Smatch and S²Match as Hard Smatch and Soft Smatch, respectively. Note that the Wu-Palmer distance based on the standard WordNet taxonomy struggles with accurately measuring the distance for verbs, adjectives, and adverbs because these parts of speech have less hierarchical structure and are sometimes unconnected. To overcome this limitation, we measure the distance with the generated encodings discussed in Section 4, enabling a more precise computation of Wu-Palmer similarity.

One of our research objectives is to make a model that is able to come up with a reasonable interpretation of a concept that it has never encountered during training. The Parallel Meaning Bank offers training, development, and test sets, all featuring similar distributions (Figure 4). The PMB data indicates that concepts are frequently used with the first word sense. This is not very surprising, because WordNet tends to list the most used senses first for each word.

This poses the following problem to meet our research objective. Say we give the model for semantic parsing a sentence with an unknown word (a word that the model hasn’t seen during training). The model will likely transform it into a WordNet concept with sense number 1, based on the statistics seen on training. Hence, the chance that the model got it correct is very high. But does such a model demonstrate some kind of semantic understanding? Not really—it just produced a pattern that it has seen many, many times during training.

In this case, we need to evaluate capability of dealing with rare or unseen word senses. We do this by creating a challenge test set consisting of more than a hundred English sentences and their gold standard meaning representations, where each sentence contains one or more words (nouns, verbs, or adjectives) that are not part of the training and development set, or are present in the training set with a different sense. To ensure an insightful evaluation, we make certain that the corresponding meaning for these unknown concepts do not correspond to the first sense. As a source of inspiration we use the glosses and example sentences found in WordNet for a particular word sense, and add enough context to disambiguate the meaning of the word. For instance, in “the moon is waxing,” we have the third sense for the verb, resulting in the concept wax.v.03, not seen in training (although wax.v.01 could be part of the training data). For each concept we construct three sentences in which the concept is expressed with enough context for a human to understand the intended meaning.

The entire challenge set comprises 500 example sentences paired with their meaning representation in SBN format, containing 430 unknown nouns, 128 unknown verbs, and 65 unknown adjectives/adverbs. We verified and manually corrected the annotations when needed to guarantee their gold-standard quality. In Table 9 in Section 6, we showcase some examples along with the predictions of different parsers.

Modern semantic parsing predominantly utilizes sequence-to-sequence models trained with linearized meaning representations (Barzdins and Gosko 2016; van Noord and Bos 2017; Bevilacqua, Blloshmi, and Navigli 2021). In our approach, we retain the sequence-to-sequence architecture but adapt the output to represent a linearized graph of semantic representation encoded with taxonomical encodings using the technique proposed by Bos (2023) as presented in Section 3.6. This output representation needs to undergo a process of interpretation (Figure 5). This interpretation is implemented by mapping the taxonomical encodings into concepts and relations of our human-readable dictionary. This dictionary consists of WordNet and the ontology of other symbols including the semantic roles. In other words, what the mapper does is translating each encoding into a traditional format found in the standard meaning representations.

The function ℳ in Equation (2) denotes the symbolic interpretation process, encapsulating the overall mapping of a meaning representation in taxonomical encodings. The function M in Equation (3) operates on individual elements of this meaning representation. It distinguishes three cases: (a) If an element t_m strictly follows the tax code format and is listed in the dictionary, it is directly mapped to the corresponding lemma-pos-sense, role, operator, or discourse relation format—for instance, n1222212113423100000000 is part of $D$⁠, and mapped to hobby.n.03; (b) If an element is not part of the dictionary but a valid taxonomical encoding, it undergoes computation by C, a traversal function that sifts through all tax codes in $D$ to identify the encoding closest to the input according to the Wu-Palmer similarity metric—for instance, n1233111111110000000000 is not part of $D$⁠, but approximated by C to n1233111111100000000000 which is part of D, and then mapped to a WordNet synset, for example, wheat_beer.n.01;⁶ (c) Elements that are not encoded are regarded as literals and left unchanged—for instance, John is kept as it is.

Hence, in our parsing system, taxonomical encodings serve as an intermediate representation, not as the final output. The interpretation component in the pipeline (Figure 5) generates a meaning graph that is readable for humans, encoded with lemma-pos-sense information and the usual labels for roles, operators, and discourse relations format (Table 1). The advantages of taxonomical encodings will be revealed in the following experiment sections.

We will compare three different representation methods for conceptual predicates: the standard one based on lemma, part of speech, and sense number (LPS, henceforth), one based on rather arbitrary WordNet identifiers (WID), and one based on our novel taxonomical encodings (TAX). The data that we use to run our experiments is drawn from the PMB. For evaluation we use both the standard test set to assess the overall semantic parsing accuracy (for English and German) as well as the challenge set dedicated to measure the ability to interpret concepts not part of the training data (for English only). Furthermore, we probe the models by extracting the sense embeddings to get an idea of whether they reflect the taxonomical information encoded in WordNet.

For our experiments, we selected the gold-standard English and German data from the PMB⁷ as detailed in Table 5. English data is divided into training, development, and test sets following an 8:1:1 split ratio, and German data follows a 4:3:3 split ratio to make the development and test sufficiently large for evaluation purposes.

In our experiments, we utilized the most frequently used seq2seq architecture, specifically leveraging the transformer-based T5 (Raffel et al. 2019) and BART (Lewis et al. 2020), two pre-trained transformer-based architectures. Specifically, we fine-tuned their multilingual variants (because we include both English and German): mT5 (Xue et al. 2021), byT5 (Xue et al. 2022), and mBART (Liu et al. 2020). mT5 builds upon the T5 model with pre-training on multi-languages corpus. byT5 enhances the multilingual approach through byte-level processing, making it especially effective at handling languages with limited data resources. Meanwhile, mBART also leverages a multilingual corpus for its pre-training phase and adopts a denoising autoencoder strategy. A noteworthy distinction between these models lies in their tokenization approaches: mT5 and mBART utilize sub-word tokenization, while byT5 uses byte-level tokenization. In our task, whether employing the lemma-pos-sense notation, WordNet identifiers, or taxonomical encodings, each notation presents a format distinct from the natural languages that were seen in their pre-training corpora. This divergence challenges the tokenization strategies of the models and their proficiency in processing new language with limited data.

We set the learning rate to 10⁻⁴, included a decay rate of 0.5, and set a patience threshold of 5 for early stopping. More details are provided in Appendix D. Given the relatively small size of the German dataset for fine-tuning (1,256 instances for training), we initially fine-tune the models on the English data before fine-tuning them with the German data. Each experiment is ran three times to calculate the average and standard deviation, which are detailed in the results tables.⁸

The results for semantic parsing for each of the three different meaning representations are presented in Table 6 (English) and Table 7 (German). We show the standard (hard) Smatch score (HSm) for exact semantic matching and the soft Smatch score (SSm) for approximate semantic matching. We also include the rate of ill-formed output (IFR), as the seq2seq architectures that we employ do not guarantee well-formed graph meaning representations (any output that is ill-formed is assigned a score of 0).

The standard Smatch scores are a little bit below earlier reported F-scores on PMB data (a Hard Smatch score 94.7 for English and 92.0 for German, as reported by Wang et al. 2023), but they can be considered decent given that we only train on the gold data part and we don’t use further pre-training on silver and bronze data from the PMB corpus. Additionally, our research objective is not to reach the highest performance, but rather compare performance of differently structured predicate symbols in meaning representations. The results for German (Table 7) are lower than those for English. We think this is caused by two factors: There is, compared to English, less typographical correspondence between the input words and output meanings, and there is less training data available for German. Nonetheless, the results for German are in line with those for English.

It is interesting to compare the performance of the three different architectures: mT5, byT5, and mBART. Despite the fact that all models are based on the seq2seq and encoder-decoder framework, they exhibit non-consistent performance on these three representations. This is due to the difference in their pre-training objectives and corpora, activation functions, parameter initialization, and other aspects (we kindly refer the reader to the original papers of these models; see Section 6.2). For instance, for both English and German, mT5, when trained using the WID representation, shows superior Hard Smatch score compared with TAX; in contrast, byT5 and mBART obtain higher Hard Smatch scores using TAX than using WID (Tables 6 and 7).

The byT5 model achieved the highest Smatch scores among all settings and both languages. We believe that the tokenization strategy of byT5 is the main reason for its good performance. This is in line with the findings of van Noord, Toral, and Bos (2020), who suggest that DRS parsing benefits from the character-level tokenization. Giving a concrete example, mT5’s sub-word tokenizer segments hobby.n.03, 101612476, and n1222212113423100000000 into disjointed chunks like [hobby,., n,.03], [10,1612,476], and [n,1222,2121,1342,31,00000000], respectively. In contrast, byT5’s byte-level tokenizer processes the same inputs into [h, o, b, b, y,., n,.,0,3], [1,0,1,6,1,2,4,7,6], and [n,1,2,2,2,1,2,1,1,3,4,2,3,1,0,0,0,0,0,0,0,0], offering a more meaningful and robust segmentation. This distinction is particularly crucial for taxonomical encodings, where each character represents a specific layer within the taxonomy.

In the rest of this discussion we will focus on byT5 given its superior performance. Based on the Hard Smatch scores, LPS emerged as the top performer, achieving scores of 87.4 for English and 80.5 for German. WID and TAX perform similarly: Both are slightly lower than LPS. We think the main reason why LPS outperforms TAX and WID with Hard Smatch is that LPS benefits from (a) generating a lemma by copying character sequences from the text input to the meaning output, and (b) generating the most frequent sense number “01” (see Section 5.2) and thereby producing the correct predicate symbol. Evidences for (a) can be found in Appendix E, where we feed misspelled words to the models revealing copying behavior for LPS. Evidences for (b) can be found in Appendix F where the majority of sense numbers chosen is “01” for LPS. The predicate symbols for TAX and WID are not based on lemmas and sense numbers (see Table 1), so they cannot “benefit” from copying lemmas or producing the most frequent sense.

However, the situation changes when we turn to the results of approximate semantic matching, where the TAX-parser demonstrates better performance (91.8 for English and 88.6 for German). The Soft Smatch score of TAX-parsers improved by a minimum of 3.2 points over Hard Smatch, reaching a peak increase of 6.2 for English and 8.5 for German. Conversely, while the Soft Smatch scores for both LPS and WID saw a modest rise, both the magnitude of their increases and final Soft Smatch scores fell short when compared to the TAX-parser. In this case, the larger increase demonstrates that TAX is doing something interesting that LPS and WID are not capable of. In Section 6.4 and 6.5, we will perform a deeper analysis of this behavior.

Considering the IFR, we can observe that there are two main causes for ill-formed outputs. One is that the index points to a non-existent concept, and the other is that the generated graph is cyclic. We found that both the WID-parser and TAX-parser significantly reduced the frequency of index-related prediction errors, which thus reduced the IFR for both English and German. These errors typically stem from the model’s limited understanding of the generated graph structure. The low IFR can be seen as evidence that proposed uniform representation using encodings enhances the seq2seq model’s comprehension of semantic graph structures.

Although the overall results for semantic parsing already favor our newly proposed taxonomical encodings (TAX), we also want to show that TAX is making fewer absurd predictions than its alternatives, LPS and WID. In Section 5.2 we presented a challenge test set for semantic parsing that contains out-of-distribution concepts. There are two ways to look at the results of the three different approaches on this stress test: globally, using Smatch scores; or locally, looking in detail on how the three approaches react on unknown concepts. The global results in terms of Hard Smatch and Soft Smatch metrics remain in line with the results of the standard tests of the previous section: For ByT5, the LPS-parser scores the highest Hard Smatch (73.3) while the TAX-parser scores the highest Soft Smatch (78.1).

For a more fine-grained analysis we looked at how the three approaches dealt with the unknown concepts. For each example sentence in the challenge test set we identified the unknown concepts and paired them with the prediction of the corresponding concepts in the output meaning representation. This was done via an automatic alignment of concepts followed by human verification and correction when needed. Then we applied the Wu-Palmer similarity to each concept pair (gold vs. predicted). The results of identifying these unknown concepts are presented in Table 8.

As Table 8 shows, none of the three approaches performs very well. Recall that achieving a perfect score on this task is highly unlikely—only by chance a parser might pick the correct word sense. Hence, all three parsers are expected to make mistakes, but there are important differences in the severity of these mistakes (Table 9).

The TAX encodings show the best performance for unknown noun and verb concepts. The standard notation following the LPS convention yields mediocre results because the parser will in most cases default to the most frequent (first or second) sense following the sense distribution in the training data. The WID parser performs surprisingly well, so the identifiers exhibit some systematic grouping that we are not aware of. Unknown verb concepts seem harder to predict, perhaps because they show less hierarchical structure in WordNet than nouns.

The LPS-parser makes the best predictions for adjectives and adverbs. This can probably be attributed to three factors. First, compared with nouns and verbs, there is no hierarchical structure in WordNet for adjectives and adverbs (see Section 4.3). Second, the Wu-Palmer measure that we use for similarity is not optimal for adjectives and adverbs, as it doesn’t take polarity into account in a principled way.⁹ Third, the training set includes only a modest number of adjectives (2,845) and adverbs (665), limiting the model to effectively learn the taxonomical information inside the encodings. In contrast, with adequate data for nouns (35,836) and verbs (8,620), the model significantly benefits from taxonomical information.

Table 9 shows some challenge set examples of unknown concept predictions for the three different parsing models (the results of the entire challenge set are shown in Appendix G). As we have seen before, the LPS-parser is extremely good at transforming word forms to a lemma and a high frequency sense number, but this strategy does not fare well on the challenge set. In fact, it makes severe mistakes, predicting thrush (the infection) instead of thrush (the bird), or Micky Mantle (the baseball player) rather than mantle (the garment). However, there were several cases where the LPS-parser had a “lucky strike,” when it picked the second sense for a lemma which happened to be correct. A case in point is Hungarian (the language), where the LPS-parser picked the second sense, perhaps because most languages in WordNet happen to be assigned the second sense (the first sense is usually the inhabitant of a country).

Most interestingly, in analogy with recent approaches to image classification (Mukherjee, Garg, and Roy 2021; Bertinetto et al. 2020), the TAX-parser “makes the best mistakes”, as it often predicted concepts similar to the unknowns. Table 9 shows some intriguing examples. For instance, when given the sentence “Jennifer cooked the bass in a steamer”, it predicts refrigerator which is close in meaning to steamer (the cooking utensil sense) as they are both appliances. For the sentence “The soldier was shot in the calf”, it predicts cheek (human face) which is close in meaning to calf (part of a human leg) as they are both body parts. And for the sentence “The woman wore a short black mantle”, it predicts coat which is close in meaning to mantle (a sleeveless garment) as they are both pieces of clothing.

In other words, the TAX-parser makes mistakes, but less drastic ones than the mistakes made by the LPS-parser, because it will attempt to find a concept that is close in meaning (exploiting the contextual understanding of the pre-trained language model) rather than copying a lemma from the textual input to output meaning, as the LPS-parser seems to be doing.

To check whether our models learn the hierarchical taxonomical information during the training process, we use a probing technique to investigate and understand the internal representations and knowledge encoded within the model. Probing is a recent method to validate whether neural models possess certain (structural) properties (Ettinger, Elgohary, and Resnik 2016; Misra, Rayz, and Ettinger 2023; Petersen and Potts 2023).

In our case, we probe the embeddings of the unknown concepts, given their three corresponding sentences in the challenge set (see Section 5.2). Because we want to compare embeddings of different levels of specificity to reflect the model’s understanding of the WordNet structure, we place each unknown concept into a small hierarchy of four levels based on itself (the most specific level) and its first three hypernyms (increasing in generality on each level). To extend coverage, we add the WordNet co-hyponyms to each concept (except for the most general level).

For instance, for the concept hobby.n.03 the first three hypernyms in WordNet are falcon.n.01, hawk.n.01, and raptor.n.01. Then we expand each of these concepts with their co-hyponyms from WordNet. For example, for hobby.n.01 we obtain the co-hoponyms gyrfalcon.n.01 and kestrel.n.01 among others, and so on. All these concepts together form what we call a concept group. Each concept group consists of four levels of concepts and is paired with three different sentence templates based on the challenge set. Each sentence template contains exactly one blank in which the lemma corresponding to the concept is filled in (hobby, kestrel, gyrfalcon, hawk, raptor, etc.). For our running example, we have the sentence template “Powerful and fast-flying, the ___ hunts medium-sized birds.”, and filling in the blank with the corresponding lemma of the concept group results in a new sentence for each concept. The sentence templates are used for all four levels in the concept group. Table 10 shows two concept groups with sentence templates.

This way, for each level in the concept group we obtain around 25 sentences. We input these sentences to the model and extract the embeddings of the concepts in the last layer of the model’s encoder. We average the embeddings of all lemmas for each sentence template and do this for each level. This gives us four embeddings for each concept, ranging from specific (e.g., hobby, gyrfalcon, …) to general (e.g., raptor).

To evaluate the reflection of the hierarchical information captured by the embeddings, we can compare the semantic distances of the embeddings representing the four levels of specificity. The assumption here is that the more generic a concept, the greater the semantic distance to a specific concept should be. We do this by computing the cosine distance for all combinations of specificity levels (with n levels, this gives us $n2$ distances). For the four levels that we have, we obtain six distance pairs, and a total of 15 comparisons to make. Each comparison is evaluated as “satisfied” (following the WordNet hierarchy) or not. For instance, the distance of a specific concept (e.g., hobby.n.03) to a slightly more generic concept (e.g., falcon.n.01) should be smaller than the distance of that concept to the most generic concept (e.g., bird.n.01).

The final score is computed by the number of satisfied comparisons divided by the total number of comparisons (Table 11). We call this metric the Hierarchy Reflection Score (HRS), and the pseudo code for the metric is shown in Appendix G. The score (a number between 0 and 1) reflects the hierarchical structure in the model: The higher the score, the closer it follows WordNet’s ontology. We distinguish two variants of HRS: base and all. The base-HRS metric only compares the distances to the most specific concept, whereas all-HRS compares all levels.¹⁰ In our experiment, we compare three different models (LPS-byT5, WID-byT5, and TAX-byT5) and pre-trained sense embeddings, SensEmBERT¹¹ by Scarlini, Pasini, and Navigli (2020a) for 130 concepts.

Among the models we trained for different semantic representations, the TAX-byT5 model achieved the highest score. The WID-byT5 model delivered a moderate performance, while the LPS-byT5 model had the lowest score. This aligns with the results we observed in the semantic parsing task and unknown concept identification task, where the TAX-byT5 model demonstrated superior structural understanding compared to the other two models. SensEmBERT demonstrates superior performance, but it is unsurprising given that it is specifically trained to adhere to the WordNet hierarchy, compared with the other three models we trained on semantic parsing.

We can also visualize the results of the probing methods. We follow the method by Lai and Nissim (2022) and apply Principal Component Analysis (PCA) to reduce the dimensionality of embeddings. Figure 6 shows the visualizations for the two example sets listed in Table 10. We use three arrows to connect the averaged embeddings of the four different specificity levels. Intuitively, the more the arrows follow a straight line in the same direction, the better they reflect the WordNet hierarchy. For instance, the embeddings of the TAX-parser succeed to reflect the WordNet hierarchy for the conceptual group for driven.n.10, as the arrows in the right of Figure 6c form a relatively straight line. The embeddings of SensEmBERT do not entirely reflect the WordNet hierarchy for almond.n.02, as the connected arrows in the left of Figure 6a show a slight turn. The embeddings of the LPS-parser fail to reflect the WordNet hierarchy driven.n.10, which can be seen by the big turn of the arrows in 6b. Although Figure 6 only shows the PCA of two instances, it does nicely illustrate the difference in interpretation of hierarchical structure of the models or the lack thereof.

We showed that by taking an existing lexical ontology, WordNet, we are able to generate hierarchical compositional encodings for predicate symbols for nouns, verbs, adjectives, and adverbs. We complemented these with encodings for semantic roles, relations, and logical operators. The resulting formal meaning representations contain concept representations that are normalized, abstracting away from specific languages. These extremely rich conceptual representations are still “parseable” for neural models. For English and German, parsing performance is a little bit lower than the standard lemma-pos-sense notation under Hard Smatch (exact semantic matching), which we attribute to the input-output copying capabilities (translation) of the neural models. However, the advantage of taxonomical encodings is evidenced by a higher Soft Smatch score (approximate semantic matching) and a superior identification of out-of-distribution concepts. Furthermore, the probing results indicate that the models trained with the taxonomical encodings exhibit superior structural understanding capabilities.

We believe that these results are encouraging and show a promising way to combine distributional semantics with formal semantics. We hope that the approach presented in this article is inspirational for future work on neural-symbolic semantic processing. We envision potential both on the symbolic and the neural side.

On the symbolic side, there is a lot of space to further explore the taxonomical encodings. The current encodings are complex, and perhaps there are ways to reduce the number of layers, or different ways of incorporating verbs and adjectives. Ontologies other than WordNet can be explored, as well as different representations of concepts (perhaps pictograms), and better methods for measuring similarity, in particular that of adjectives and adverbs (especially for the case of antonymy). Another direction for future work is exploring alternative evaluation metrics to better handle the complexity introduced by the fine-grained similarity evaluation in Soft Smatch. The Soft Smatch method in this article relies on the hill-climbing algorithm of Hard Smatch, which can sometimes result in unwanted matches (Opitz 2023).¹²

On the neural side, there are many potential areas worth exploring that have fallen outside the scope of this article. First, incorporating sense embeddings is promising, but integrating them into a neural semantic parser that produces complete meaning representations is challenging.¹³ Another interesting area of research is to investigate modifications of the loss function aimed at enhancing the model’s understanding of taxonomical information in the encodings, where weights are assigned to characters based on their positions. Another direction to consider is using large language models such as Phi3, Mistral, LlaMa, and GPT-4 for semantic parsing, as they are known to have strong language modeling capabilities. However, their architectures (decoder-only) and corresponding experimental settings strongly differ from those of our models. Some pilot experiments that we ran indicate that their performance, whether using standard representations or taxonomical encodings, is by far inferior to our models. An investigation on why large language models perform so poorly on semantic parsing goes beyond the objectives of this article but is perhaps an exciting topic for future research.

We show here the three different types of meaning representations used in our experiments. We do this for the text “John, a keen birdwatcher, was delighted to see a hobby.” in graph format for readability. In the experiments we use the sequence notation.

Figure B.1 shows the hierarchy of roles and relations as used in the Parallel Meaning Bank. This hierarchy is an extension of the one proposed for VerbNet (Bonial et al. 2011).

The identifier and character encoding of operators and discourse relations in Table C.1 and Table C.2.

To facilitate reproduction, we detail the important hyperparameters used. For the WID and TAX encodings, we adopt a larger patience and decay rate, allowing ample time for convergence. This decision stems from experimental observations indicating that these novel (to the models) representations exhibit a slower convergence rate compared to LPS. We test using hierarchical loss, which give higher weight to the characters on the left within each word, but the initial experiments didn’t show any improvements.

In order to assess the ability of models to deal with misspellings we created a test suite of English sentences paired with meaning representations where each sentence contained a commonly misspelled content word, that is acceptible, humourous, enterpreneur. The results, shown in Table E.1, demonstrate that the traditional lemma-pos-sense notation fails to identify wrongly spelled content words caused by its tendency to copy character sequences from input words to output lemmas.

This appendix shows predictions of the three semantic parsers (LPS, WID, TAX) on the challenge set for nouns (Table F.1), verbs (Table F.2), and modifiers (Table F.3). The challenge set includes several sentences for the out-of-vocabulary concepts. For reasons of space, only one prediction for each concept is shown. The complete predictions are available on our GitHub site.¹⁴

Algorithm 1 shows the calculation of Hierarchy Reflection Score (HRS-all). To be more detailed, in our four specificity levels, for the inequalities d(0,1) < d(0,2), d(0,1) < d(0,3), d(0,2) < d(0,3), d(1,2) < d(0,2), d(1,2) < d(1,3), d(1,3) < d(2,3), d(2,3) < d(0,3), HRS is $17$ if one of them is satisfied; HRS is $27$ if two of them is satisfied;…; HRS is $77$ if all of them are satisfied.

We wish to thank the three anonymous reviewers for their helpful suggestions that greatly improved this article. We would also like to express our gratitude to Huiyuan Lai, Gertjan van Noord, and Juri Opitz for their valuable feedback on earlier versions of our work.