Weisfeiler-Leman in the Bamboo: Novel AMR Graph Metrics and a Benchmark for AMR Graph Similarity

Meaning representations aim at capturing the meaning of text in an explicit graph format. A prominent framework is abstract meaning representation (AMR), proposed by Banarescu et al. (2013). AMR views sentences as rooted, directed, acyclic, labeled graphs. Their nodes are variables, attributes, or (open-class) concepts and are connected with edges that express semantic relations.

There are many use cases in which we need to compare or relate two AMR graphs. A common situation is found in parser evaluation, where AMR metrics are widely applied (May, 2016; May and Priyadarshi, 2017).¹ Yet, there are more situations where we need to measure similarity of meaning as expressed in AMR graphs. For example, Bonial et al. (2020) leverage AMR metrics in a semantic search engine for COVID-19 queries, Naseem et al. (2019) use metric feedback to reinforce AMR parsers, Opitz (2020) emulates metrics for referenceless AMR ranking and rating, and Opitz and Frank (2021) use AMR metrics for NLG evaluation.

So far, multiple AMR metrics (Cai and Knight, 2013; Cai and Lam, 2019; Song and Gildea, 2019; Anchiêta et al., 2019; Opitz et al., 2020) have been proposed to assess AMR similarity. However, due to a lack of an appropriate evaluation benchmark, we have no empirical evidence that could tell us more about their strengths and weaknesses or offer insight about which metrics may be preferable over others in specific use cases.

Additionally, we would like to move beyond the aforementioned metrics and develop new metrics that account for graded similarity of graph substructures, which is not an easy task. However, it is crucial when we need to compare AMR graphs in a deeper way. Consider Figure 1, which shows two AMRs that convey very similar meanings. All aforementioned metrics assign this pair a low similarity score, and—if alignment-based, as is Smatch (Cai and Knight, 2013)—find only subpar alignments.² In this case, we want a metric that provides us with a high similarity score and, ideally, an explanatory alignment.

The structure of this paper is as follows. In §2 we discuss related work. In §3 we describe our first contribution: new AMR metrics that aim at unifying the strengths of previous metrics while mitigating their weaknesses. Specifically, our new metrics are capable of matching larger substructures and provide valuable n:m alignments in polynomial time. In §4 we introduce Bamboo, our second contribution: It is the first benchmark data set for AMR metrics and includes novel robustness objectives that probe the behavior of AMR metrics under meaning-preserving and meaning-altering transformations of the inputs (§5). In §6 we use Bamboo for a detailed, multi-faceted empirical study of previous and our proposed AMR metrics.

We release Bamboo and our new metrics.³

The ‘canonical’ and widely applied AMR metric is Smatch (Semantic match) (Cai and Knight, 2013). It solves an NP-hard graph alignment problem approximately with a hill-climber and scores matching triples. Smatch has been adapted to S²match (Soft Semantic match), by Opitz et al. (2020) to account for graded similarity of concept nodes (e.g., cat—kitten), using word embeddings. Smatch has also been adapted by Cai and Lam (2019) in W(eighted)Smatch (WSmatch), which penalizes errors relative to their distance to the root. This is motivated by the hypothesis that “core semantics” tend to be located near a graph’s root.

Recently, two new AMR metrics have been proposed: Sema by Anchiêta et al. (2019) and SemBleu by Song and Gildea (2019). Common to both is a mechanism that traverses the graph. Both start from the root, and collect structures with a breadth-first traversal (BFS). Also, both ablate the variable alignment of (W)S⁽²⁾match and only consider their attached concepts, which increases computation speed. Apart from this, the metrics differ significantly: SemBleu extracts bags of k-hop paths (k≤3) from the AMR graphs and thereupon calculates BLEU (Papineni et al., 2002). Sema, on the other hand, is somewhat simpler and provides us with an F1 score that it achieves by comparing extracted triples.

Most AMR metrics have been designed for semantic parser evaluation, and therefore determine a score for structure overlap. While this is legitimate, with extended use cases for AMR metrics arising, there is increased awareness that structural matching of labeled nodes and edges of an AMR graph is not sufficient for assessing the meaning similarity expressed by two AMRs (Kapanipathi et al., 2021). This insufficiency has also been observed in cross- lingual AMR parsing evaluation (Blloshmi et al., 2020; Sheth et al., 2021; Uhrig et al., 2021), but is most prominent when attempting to compare the meaning of AMRs that represent different sentences (Opitz et al., 2020; Opitz and Frank, 2021). This work argues that in cases like Figure 1, the available metrics do not sufficiently reflect the similarity of the two AMRs and their underlying sentences.

STS (Baudiš et al., 2016a, 2016b; Cer et al., 2017) and SICK (Marelli et al., 2014) elicited human ratings of sentence similarity on a Likert scale. While STS annotates semantic similarity, SICK annotates semantic relatedness. These two aspects are highly related, but not the exact same (Budanitsky and Hirst, 2006; Kolb, 2009). Only the highest scores on the Likert scales of SICK and STS can be seen as reflecting the equivalence of meaning of two sentences. Other data sets contain binary annotations of paraphrases (Dolan and Brockett, 2005), that cover a wide spectrum of semantic phenomena.

Metric benchmarking is an active topic in NLP research and led to the emergence of metric benchmarks in various areas, most prominently MT and NLG (Gardent et al., 2017; Zhu et al., 2018; Ma et al., 2019). These benchmarks are useful since they help to assess and select metrics and encourage their further development (Gehrmann et al., 2021). However, there is currently no established benchmark that defines a ground truth of graded semantic similarity between pairs of AMRs, and how to measure it in terms of their structural representations. Also, we do not have an established ground truth to assess what alternative AMR metrics such as (W—S²)match or SemBleu really measure, and how their scores correlate with human judgments of the semantic similarity of sentences represented by AMRs.

Previous AMR metrics have complementary strengths and weaknesses. Therefore, we aim to propose new AMR metrics that are able to mitigate these weaknesses, while unifying their strengths, aiming at the best of all worlds. We want:

• i)

  an interpretable alignment (Smatch);

• ii)

  a fast metric (Sema, SemBleu);

• iii)

  matching larger substructures (SemBleu);

• iv)

  and assessment of graded similarity of AMR subgraphs(extending S²match).

This section proposes to make use of the Weisfeiler-Leman graph kernel (WLK) (Weisfeiler and Leman, 1968; Shervashidze et al., 2011) to assess AMR similarity. The idea is that WLK provides us with SemBleu-like matches of larger sub-structures, while bypassing potential biases induced by the BFS-traversal (Opitz et al., 2020). We then describe the Wasserstein Weisfeiler Leman kernel (WWLK) (Togninalli et al., 2019) that is similar to WLK but provides (i) an alignment of atomic and non-atomic substructures (going beyond Smatch) and (ii) a graded match of substructures (going beyond S²match). Finally, we further adapt WWLK to WWLK_Θ, a variant that we tailor to learn semantic edge parameters, to better assess AMR graphs.

The Weisfeiler-Leman kernel (WLK) method (Shervashidze et al., 2011) derives sub-graph features from two input graphs. WLK has shown its power in many tasks, ranging from protein classification to movie recommendation (Togninalli et al., 2019; Yanardag and Vishwanathan, 2015). However, so far, it has not been applied to (A)MR graphs. In the following, we describe the WLK method.

Specifically, we use the cosine similarity kernel and two iterations (K = 2), which implies that every node receives information from its neighbors and their immediate neighbors. For simplicity we will first treat edges as undirected, but later will experiment with various directionality parameterizations.

S²match differs from all other AMR metrics in that it accepts close concept synonyms for alignment (up to a similarity threshold). But it comes with a restriction and a downside: i) it cannot assess graded similarity of (non-atomic) AMR subgraphs, which is crucial for assessing partial meaning agreement between AMRs (as illustrated in Figure 1), and ii) the alignment is costly to compute.

We hence propose to adopt a variant of WLK, the Wasserstein-Weisfeiler Leman kernel (WWLK) (Togninalli et al., 2019), for the following two reasons: (i) WWLK can assess non-atomic subgraphs on a finer level, and (ii) it provides graph alignments that are faster to compute than any of the existing Smatch metrics: (W)S⁽²⁾match.

WWLK works in two steps: (1) Given its initial node embeddings, we use WL to project the graph into a latent space, in which the final node embeddings describe varying degrees of contextualization. (2) Given a pair of such (WL) embedded graphs, a transportation plan is found that describes the minimum cost of transforming one graph into the other. In the top graph of Figure 3, f indicates the first step, while Wasserstein distance indicates the second. Now, we describe the steps in closer detail.

The WL-embedding mechanism of WWLK (Eq. 4) associates a weight w(u, v) ∈ℝ with each edge. For unlabeled graphs, w(u, v) is simply set to one. To distinguish between the discrete AMR edge labels, in WWLK we have used random weights. However, AMR edge labels encode complex relations between nodes, and simply choosing random weights may not be enough. In fact, we hypothesize that different edge labels may impact the meaning similarity of AMR graphs in different ways. Whereas a modifier relation in an AMR graph configuration may or may not have a significant influence on the overall AMR graph similarity, an edge representing negation is bound to have a significant influence on the similarity of different AMR graphs. Consider the example in Figure 3: In the top figure, we embed AMRs for The pretty warbler sings and The bird sings gently, which have similar meanings. In the bottom figure, the second AMR has been changed to express the meaning of The bird doesn’t sing, which clearly reduces the meaning similarity of the two AMRs. Hence, we hypothesize that learning edge parameters for different AMR relation types may help to better adjust the graph embeddings, such that the Wasserstein distance may increase or decrease, depending on the specific meaning of AMR relation labels, and thus to better capture global meaning differences between AMRs (as outlined in Figure 3: f_θ).

Formally, to make the Wasserstein Weisfeiler- Leman kernel better account for edge-labeled AMR graphs, we learn a parameter set Θ that consists of parameters θ^edgeLabel, where edgeLabel indicates the semantic relation, i.e., edgeLabel ∈ L = {:arg0, :arg1, …, :polarity, …}. Hence, in Eq. 4, we can set w(u, v) = θ^{label(u, v)} and apply multiplication θ^{label(u, v)} ⋅ x(u)^k. To facilitate the multiplication, we either may learn a matrix Θ ∈ℝ^|L|×d or a parameter vector Θ ∈ R^|L|. In this paper, we constrain ourselves to the latter setting, that is, our goal is to learn a parameter vector Θ ∈ R^|L|.

To find suitable edge parameters Θ, we propose a zero^th order (gradient-free [Conn et al., 2009]) optimization setup, which has the advantage that we can explicitly teach our metric to better correlate with human ratings, optimizing the desired correlation objective without detours. In our case, we apply a simultaneous perturbation stochastic approximation (SPSA) procedure to estimate gradients (Spall, 1987, 1998; Wang, 2020).⁵

We now describe the creation of Bamboo, which aims to provide the first benchmark that allows researchers to empirically (i) assess AMR metrics, (ii) compare AMR metrics, and possibly (iii) train AMR metrics.

Accordingly, we select, as evaluation targets for AMR metrics, three notions of sentence similarity, which have previously been operationalized in terms of human-rated evaluation datasets: (i) the semantic textual similarity (STS) objective from Baudiš et al. (2016a, b); (ii) the sentence relatedness objective (SICK) from Marelli et al. (2014); (iii) the paraphrase detection objective (PARA) by Dolan and Brockett (2005).

Each of these three evaluation data sets can be seen as a set of pairs of sentences (s_i,s_i′) with an associated score humanScore(⋅) that provides the human sentence relation assessment score reflecting semantic similarity (STS), semantic relatedness (SICK) and whether sentences are paraphrastic (PARA). Hence, each of these data sets can be described as {(s_i,s_i′, humanScore$(si,si′)=yi)}i=1n$⁠. Both STS and SICK offer scores on Likert scales, ranging from equivalence (max) to unrelated (min), while PARA scores are binary, judging sentence pairs as being paraphrases (1), or not (0). We min-max normalize the Likert scale scores to the range [0,1] to facilitate standardized evaluation.

For Bamboo, we replace each pair (s_i,s_i′) with their AMR parses: (p_i = parse(s_i), p_i′ = parse(s_i′)), transforming the data into ${(pi,pi′,yi)}i=1n$⁠. This provides the main partition of the benchmarking data for Bamboo, henceforth denoted as Main.⁶ Statistics of Main are shown in Table 1). The sentences in PARA are longer compared to STS and SICK. The corresponding AMR graphs are, on average, much larger in number of nodes, but less complex with respect to the average density.⁷

We choose a strong parser that achieves high scores in the range of human- human inter-annotator agreement estimates in AMR banking: The parser yields 0.80–0.83 Smatch F1 on AMR2 and AMR3. The parser, henceforth denoted as T5S2S, is based on an AMR fine-tuned T5 language model (Raffel et al., 2019) and produces AMRs in a sequence-to-sequence fashion.⁸ It is on par with the current state-of-the-art that similarly relies on seq-to-seq (Xu et al., 2020), but the T5 backbone alleviates the need for massive MT pre-training. To obtain a better picture of the graph quality we perform manual quality inspections.

From each data set (SICK, STS, PARA) we randomly select 100 sentences and create their parses with T5S2S. Additionally, to establish a baseline, we also parse the same sentences with the GPLA parser of Lyu and Titov (2018), a neural graph prediction system that uses latent alignments (which reports 74.4 Smatch score on AMR2). This results in 300 GPLA parses and 300 T5S2S parses. A human annotator⁹ inspects the (shuffled) sample and assigns three-way labels: flawed—an AMR contains critical errors that distort the meaning significantly; silver—an AMR contains small errors that can potentially be neglected; gold—an AMR is acceptable.

Results in Table 2 show that the quality of T5S2S parses is substantially better than the baseline in all three data sets. The percentage of excellent parses increases considerably (STS: +11pp, SICK: +10pp, PARA: +11pp) while the percentage of flawed parses drops notably (STS: −15pp, SICK: −8pp, PARA: −23pp). The increases in gold parses and decreases in flawed parses are significant in all data sets (p < 0.05, 10,000 bootstrap samples of the sample means).¹⁰

Besides benchmarking AMR metric scores against human ratings, we are also interested in assessing a metric’s robustness under meaning-preserving and -altering graph transformations. Assume we are given any pair of AMRs from paraphrases. A small change in structure or node content can lead to two outcomes: The graphs still represent paraphrases, or they do not. We consider a metric to be robust if its ratings correctly reflect such changes.

Specifically, we propose three transformation strategies. (i) Reification (Reify↯), which changes the graph’s surface structure, but not its meaning; (ii) Concept synonym replacement (Syno↯), which also preserves meaning and may or may not change the graph surface structure; (iii) Role confusion (Arg↯), which applies small changes to the graph structure that do not preserve its meaning.

Here, we iterate over AMR concept nodes. For any node that involves a predicate from PropBank, we consult a manually created database of (near-) synonyms that are also contained in PropBank, and sample one for replacement. For example, some sense of fall is near-equivalent to a sense of decrease (car prices fell/decreased). For concepts that are not predicates we run an ensemble of four WSD solvers¹² (based on the concept and the sentence underlying the AMR) to identify its WordNet synset. From this synset we sample an alternative lemma.¹³ If an alternative lemma consists of multiple tokens where modifiers precede the noun, we replace the node with a graph- substructure. So, if the concept is man and we sample adult_male, we expand ‘instance(x, man)’ with ‘mod(x, y) ∧ instance(y, adult) ∧ instance =(x, male)’. Data processing statistics are shown in Table 3.

A naïve AMR metric could be one that treats an AMR as a bag-of-nodes, omitting structural information, such as edges and edge-labels. Such metrics could exhibit misleadingly high correlation scores with human ratings, solely due to a high overlap in concept content.

In some cases g may create a meaning that still makes sense (The tiger bites the snake.$→$The snake bites the tiger.), while in others, g may induce a non-sensical meaning (The tiger jumps on the rock.$→$The rock jumps on the tiger.). However, this is not our primary concern, since in all cases, applying g achieves our main goal: It returns a different meaning that turns a paraphrase-relation between two AMRs into a non-paraphrastic one.

To implement Arg↯, for each data set (PARA, STS, SICK) we create one new data subset. First, (i) we collect all paraphrases from the initial data (in SICK and STS these are pairs with maximum human score).¹⁴ (ii) We iterate over the AMR pairs $(G,G′)$ and randomly select the first or second AMR from the tuple. We then collect all n nodes with more than one outgoing edge. If n = 0, we skip this AMR pair (the pair will not be contained in the data). If n > 0, we apply the meaning altering function g and randomly flip edge labels. Finally, we add the original $(G,G′)$ to our data with the label paraphrase, and the altered pair $(G,g(G′))$ with the label non-paraphrase (cf. Figure 5). Per the graph, we allow a maximum of 3 role confusion operations (see Table 3 for processing statistics).

We have proposed three challenging robustness objectives. Reify↯ changes the graph structure, but preserves the meaning. Arg↯ keeps the graph structure (modulo edge labels) while changing the meaning. Syno↯ changes node labels and possibly the graph structure and aims at preserving the meaning.

Reify↯ and Arg↯ are fully safe: they are well defined and are guaranteed to fulfill our goal (Eq. 12 and 17): meaning-preserving or -altering graph transforms. Syno↯ is more experimental and has (at least) three failure modes. In the first mode, depending on context, a human similarity judgments could change when near-synonyms are chosen (sleep $→$ doze, a young cat $→$ kitten, etc.). The second mode occurs when WSD commits an error (e.g., minister (political sense) $→$ priest). A third mode are societal biases found in WordNet (e.g., the node girl may be mapped onto its ‘synonym’ missy). The third mode may not really be a failure, since it may not change the human rating, but, nevertheless, it may be undesirable.

In conclusion, Reify↯ and Arg↯ confusion constitute safe robustness challenges, while results on Syno↯ have to be taken with a grain of salt.

We believe that a key benefit of the robustness challenges lies in their potential to provide complementary performance indicators, in addition to evaluation on the Main partition of Bamboo (cf. §4). In particular, the challenges may serve to assess metrics more deeply, uncover potential weak spots, and help select among metrics, for example, when performance differences on Main are small. In this work, however, the complementary nature of Reify↯, Syno↯ or Arg↯ versus Main is only reflected in the name of the partitions, and in our experiments, we consider all partitions equally. Future work may deviate from this setup.

Our proposed robustness challenges are also by no means exhaustive, and we believe that there is ample room for developing more challenges (extendingBamboo) or experimenting with different setups of our challenges (varyingBamboo¹⁵ ). For these reasons, it is possible that future work may justify alternative or enhanced setups, extensions and variations of Bamboo.

Bamboo allows us to address several open questions: The first set of questions aims to gain more knowledge about previously released metrics. That is, we would like to know: What semantic aspects of AMR does a metric measure? If a metric has hyper-parameters (e.g., SemBleu), which hyper-parameters are suitable (for a specific objective)? Does the costly alignment of Smatch pay off, by yielding better predictions, or do the faster alignment-free metrics offer a ‘free-lunch’? A second set of questions aims to evaluate our proposed novel AMR similarity metrics, and to assess their potential advantages.

We evaluate all metrics on the test set of Bamboo. The two hyper-parameters of S²match, that determine when concepts are similar, are set with a small search on the development set (by contrast, S²match_default denotes the default setup). WWLK_θ is trained with batch size 16 on the training data. S²match, WWLK and WWLK_θ all make use of GloVe embeddings (Pennington et al., 2014).

Our main evaluation metric is Pearson’s ρ between a metric’s output and the human ratings. Additionally, we consider two global performance measures to better rank AMR metrics: the arithmetic mean (amean) and the harmonic mean (hmean) over a metric’s results achieved in all tasks. Hmean is always ≤ amean and is driven by low outliers. Hence, a large difference between amean and hmean serves as a warning light for a metric that is extremely vulnerable in a specific task.

Table 4 shows AMR metric results on Bamboo across all three human similarity rating types (STS, SICK, PARA) and our four challenges: Main represents the standard setup (cf. §4), whereas Reify↯, Syno↯, and Arg↯ test the metric robustness (cf. §5).

Smatch, our baseline metric, provides strong results across all tasks (Table 4, amean: 51.28). With default parameters, S²match_default performs slightly worse on the main data for STS and SICK, but improves upon Smatch on PARA, achieving a slight overall improvement with respect to hmean (+0.30), but not amean (−0.37). S²match is more robust against Syno↯ (e.g., +4.6 on Syno↯ STS vs. Smatch), and when confronted with reified graphs (Reify↯ STS +3.3 vs. Smatch).

Finally, S²match, after setting its two hyper- parameters with a small search on the development data¹⁶ , consistently improves upon Smatch over all tasks (amean: +0.94, hmean: +1.57).

The hypothesis underlying WSmatch is that concepts that are located near the top of an AMR have a higher impact on AMR similarity ratings. Interestingly, WSmatch mostly falls short of Smatch, offering substantially lower performance on all main tasks and all robustness checks, resulting in reduced overall amean and hmean scores (e.g., main STS: −5.39 vs. Smatch, amean: −2.8 vs. Smatch, hmean: −2.9 vs. Smatch). This contradicts the ‘core-semantics’ hypothesis and provides novel evidence that semantic concepts that influence human similarity ratings are not necessarily located close to AMR roots.¹⁷

Next, we find that Sema achieves lower scores in almost all categories, when compared with Smatch (amean: −4.99, hmean −5.65), ending up at rank 7 (according to hmean and amean) among prior metrics. It is similar to Smatch in that it extracts triples from graphs, but differs by not providing an alignment. Therefore, it can only loosely model some phenomena, and we conclude that the increase in speed comes at the cost of a substantial drop in modeling capacity.

Results for SemBleu show that it is very sensible to parameterizations of k. Notably, k = 1, which means that the method only extracts bags of nodes, achieves strong results on SICK and STS. On PARA, however, SemBleu is outperformed by S²match, for all settings of k (best k (k = 2): −2.8 amean, −4.7 hmean). Moreover, all variants of SemBleu are vulnerable to robustness checks. E.g., k = 2, and, naturally, k = 1 are easily fooled by Arg↯, where performance drops massively. k = 4, on the other hand, is most robust against Arg↯, but overall it falls behind k = 2.

Since SemBleu is asymmetric, we also re- compute the metric in a ‘symmetric’ way by averaging the metric result over different argument orders. We find that this can slightly increase its performance ([k, amean, hmean]: [1, +0.8, +0.6]; [2, +0.5, +0.4]; [3, +0.2, +0.2]; [4, +0.1, +0.0]).

In sum, our conclusions concerning SemBleu are: (i) SemBleu_{k =1} (but not SemBleu_{k =3}) performs well when measuring similarity and relatedness. However, SemBleu_{k =1} is naïve and easily fooled (Arg↯). (ii) Hence, we recommend k = 2 as a good tradeoff between robustness and performance, with overall rank 4 (amean) and 6 (hmean).¹⁸

We now discuss results of our proposed metrics based on the Weisfeiler-Leman Kernel.

First, we visit the classic Weisfeiler-Leman kernel. Like SemBleu and Sema, the (alignment-free) method is very fast. However, it outperforms these metrics in almost all tasks (score difference against second best alignment-free metric: ([a∣h]mean: +1.6, +1.5) but falls behind alignment-based Smatch ([a∣h]mean: −0.8, −3.2). Specifically, WLK proves robust against Reify↯ but appears more vulnerable against Syno↯ (−5 points on STS and SICK) and Arg↯ (notably PARA, with −10 points).¹⁹

The better performance, compared to SemBleu and Sema, may be due to the fact that WLK (unlike SemBleu and Sema) does not perform BFS traversal from the root, which may reduce biases.

Basic WWLK exhibits strong performance on SICK (ranking second on main and first on Reify↯). However, it has large vulnerabilities, as exposed by Arg↯, where only SemBleu_{k =1} ranks lower. This can be explained by the fact that WWLK (7.2 Pearson’s ρ on PARA Arg↯) only weakly considers the semantic relations (whereas SemBleu_{k =1} does not consider semantic relations in the first place).

WWLK_Θ, our proposed algorithm for edge label learning, mitigates this vulnerability (29.6 Pearson’s ρ on PARA Arg↯, 1^st rank). Learning edge labels also helps assessing similarity (STS) and relatedness (SICK), with substantial improvements over standard WWLK and Smatch (STS: 66.94, +3.9 vs. WWLK and +10.6 vs. Smatch; SICK +2.1 vs. WWLK and +8.4 vs. Smatch).

In sum, WWLK_θ occupies rank 1 of all considered metrics (amean and hmean), outperforming all non-alignment based metrics by large margins (amean +4.5 vs. WLK and +6.0 vs. SemBleu_{k =2}; hmean +5.9 vs. WLK and +8.1 vs. SemBleu_{k =2}), but also the alignment-based ones, albeit by lower margins (amean +2.7 vs. S²match; hmean + 1.2 vs. S²match).

How does setting the number of iterations in Weisfeiler-Leman affect predictions? Table 5 shows K = 2 is a good choice for all WLK variants. K = 3 slightly increases performance in the latent variants (WWLK: +0.4 amean; WWLK_θ: +0.3 amean), but lowers performance for the fast symbolic matching WLK (−2.8 amean). This drop is somewhat expected: K > 2 introduces much sparsity in the symbolic WLK feature space.

Even though AMR defines directional edges, for optimal similarity ratings, it was not a-priori clear in which directions the node contextualization should be restricted when attempting to model human similarity. Therefore, so far, our WLK variants have treated AMR graphs as undirected graphs (⁠$↔$⁠). In this experiment, we study three alternate scenarios: ‘TOP-DOWN’ (forward, $→$⁠), where information is only passed in the direction that AMR edges point at and ‘BOTTOM-UP’ (backwards, $←$⁠), where information is exclusively passed in the opposite direction, and 2WAY (⁠$⇆$⁠), where information is passed forwards, but for every edge edge(x, y) we insert an edge⁻¹(y, x). 2WAY facilitates more node interactions than either TOP- DOWN or BOTTOM-UP, while preserving directional information.

Our findings in Table 6 show a clear trend: treating AMR graphs as graphs with undirected edges offers better results than TOP–DOWN (e.g., WWLK −1.6 amean; −6.6 hmean) and considerably better results when compared to WLK in BOTTOM–UP mode (e.g., WWLK −3.7 amean; −18.9 hmean). Overall, 2WAY behaves similarly to the standard setup, with a slight improvement for WWLK_θ. Notably, the symbolic WLK variant, that does not use word embeddings, appears more robust in this experiment and differences between the three directional setups are small.

Initial quality analyses (§4) suggested that the quality of Bamboo is high, with a large proportion of AMR graphs that are of gold or silver quality. In this experiment, we study how metric rankings and predictions could change when confronted with AMRs corrected by humans. From every data set, we randomly sample 50 AMR graph pairs (300 AMRs in total). In each AMR, the human annotator searched for mistakes, and corrected them.²⁰

We study two settings. (i) Intra metric agreement (IMA): For every metric, we calculate the correlation of its predictions for the initial graph pairs versus the predictions for the graph pairs that are ensured to be correct. Note that, on one hand, a high IMA for all metrics would further corroborate the trustworthiness of Bamboo results. However, on the other hand, a high IMA for a single metric cannot be interpreted as a marker for this metric’s quality. That is, a maximum IMA (1.0) could also indicate that a metric is completely insensitive to the human corrections. Furthermore, we study (ii) Metric human agreement (MHA): Here, we correlate the metric scores against human ratings, once when fed the fully gold-ensured graph pairs and once when fed the standard graph pairs. Both measures, IMA, and IAA, can provide us with an indicator of how much metric ratings would change if Bamboo would be fully human corrected.

Results are shown in Table 7. All metrics exhibit high IMA, suggesting that potential changes in their ratings, when fed gold-ensured graphs, are quite small. Furthermore, on average, all metrics tend to exhibit slightly better correlation with the human when computed on the gold-ensured graph pairs. However, supporting the assessment of IMA, the increments in MHA appear small, ranging from a minimum increment of +0.3 (SemBleu) to a maximum increment of +2.8 (S²match), whereas WWLK yields an increment of +1.8. Generally, while this assessment has to be taken with a grain of salt due to the small sample size, it overall supports the validity of Bamboo results.

We can group metrics for graph-based meaning representations into whether they compute an alignment between AMRs or not (Liu et al., 2020). A computed alignment, as in Smatch, has the advantage that it lets us assess finer-grained AMR graph similarities and divergences, by creating and exploiting a mapping that shows which specific substructures of two graphs are more or less similar to each other. On the other hand, it was still an open question whether such an alignment is worth its computational cost and enhances similarity judgments.

Experiments on Bamboo provide novel evidence on this matter: alignment-based metrics may be preferred for better accuracy. Non- alignment based metrics may be preferred if speed matters most. The latter situation may occur, for example, when AMR metrics must be executed over a large cross-product of parses (for instance, to semantically cluster sentences from a corpus). For a balanced approach, WWLK_Θ offers a good trade-off: polynomial-time alignment and high accuracy.

Figure 6 (top) shows an example where the human-assigned similarity score is relatively low (rank 1164 of 1379). Due to the graphs having the same structure (xarg0y; x arg1z), the previous metrics (except Sema) tend to assign similarities that are relatively too high. In particular, S²match finds the exact same alignments in this case, but cannot assess the concept-relations more deeply. WWLK yields more informative alignments since they explain its decision to assign a more appropriate lower rank (1253 of 1379): Substantial work is needed to transport, for example, carry-01 to slice-01.

Figure 6 (bottom) shows that WWLK produces valuable n:m alignments (play-11 vs. make-01 and music), which are needed to properly reflect similarity (note that Smatch, WSmatch, and S²match only provide 1-1 alignments). Yet, the example also shows that there is still a way to go. While humans assess this near-equivalence easily, providing a relatively high score (rank 331 of 4972), all metrics considered in this paper, including ours, assign relative ranks that are too low (WWLK: 2624). Future work may incorporate external PropBank (Palmer et al., 2005) knowledge into AMR metrics. In PropBank, sense 11 of play is defined as equivalent to making music.

Our contributions in this work are three-fold: (i) We propose a suite of novel Weisfeiler-Leman AMR similarity metrics that are able to reconcile a performance conflict between precision of AMR similarity ratings and the efficiency of computing alignments. (ii) We release Bamboo, the first benchmark that allows researchers to assess AMR metrics empirically, setting the stage for future work on graph-based meaning representation metrics. (iii) We showcase the utility of Bamboo, by applying it to profile existing AMR metrics, uncovering hitherto unknown strengths or weaknesses, and to assess the strengths of our newly proposed metrics that we derive and further develop from the classic Weisfeiler-Leman Kernel. We show that through Bamboo we are able to gain novel insight regarding suitable hyperparameters of different metric types, and to gain novel perspectives on how to further improve AMR similarity metrics to achieve better correlation with the degree of meaning similarity of paired sentences, as perceived by humans.

We are grateful to three anonymous reviewers and Action Editor Yue Zhang for their valuable comments that have helped to improve this paper. We are also thankful to Philipp Wiesenbach for giving helpful feedback on a draft of this paper. This work has been partially funded by the DFG through the project ACCEPT as part of the Priority Program “Robust Argumentation Machines” (SPP1999).