DiBiMT: A Gold Evaluation Benchmark for Studying Lexical Ambiguity in Machine Translation Open Access

Over the course of the last few decades, the field of Machine Translation (MT) has witnessed remarkable advances in terms of fluency and idiomaticity of output translations as well as model efficiency, largely due to the development of the Transformer architecture and the attention mechanism (Vaswani et al. 2017). Recently, with the scaling of model size and the increased availability of data, Large Language Models (LLMs) are bringing a breeze of change to virtually all Natural Language Processing (NLP) tasks, by combining the extraordinary representational power of modern language modeling techniques with in-context learning (Dong et al. 2024). Specifically, in MT, LLMs have demonstrated remarkable translation capabilities in several language combinations (Brown et al. 2020; Zhu et al. 2024), despite weaker performance on less-represented languages (Kocmi et al. 2023).

Notwithstanding such advancements, current systems still struggle when dealing with specific linguistic phenomena, among which lexical ambiguity poses one of the greatest challenges (Emelin, Titov, and Sennrich 2020; Campolungo et al. 2022; Iyer et al. 2023). Considered one of the hardest problems in NLP, lexical ambiguity is a pervasive linguistic phenomenon in which a given word can express different meanings depending on the context in which it occurs (Krovetz and Croft 1992). Unlike other forms of linguistic ambiguity that can arise at different levels (e.g., the morphological, syntactic, or pragmatic ones), lexical ambiguity affects individual words or expressions and their meanings.

Dealing with this problem properly is of vital importance to ensure high-quality output translations. In order to illustrate the impact of lexical ambiguity on the translation process, let us consider the examples reported in Table 1, in which we show the automatic translations of two source sentences from English into four target languages, namely, German (DE), Italian (IT), Russian (RU), and Spanish (ES), obtained with two different systems: DeepL,¹ a state-of-the-art commercial MT system, and ChatGPT,² a popular chatbot based on GPT-3.5.³ Both source sentences contain an ambiguous word highlighted in violet, namely, the noun and the verb , respectively. We identify the corresponding translations of the ambiguous source words and mark them in green with a straight underline or red with a wavy underline, depending on whether they are to be considered correct or not. For instance, the first source sentence An example is the best of instruction is translated into Italian with Un esempio è la miglioredi istruzione. In this context, the ambiguous word indicates how a specific goal is achieved, while the Italian word agenzia does not express the same meaning, resulting in an incorrect translation. In this case, correct choices would instead be maniera, mezzo, or modo. Similarly, the second source sentence the roof of a hut could be translated as il tetto di una capanna. Instead, this source text is incorrectly rendered with del tetto di una capanna, in which the expression sbucciare la corteccia literally means peel the bark.

In order to produce high-quality translations, MT systems must be able to determine the correct meaning of ambiguous source words and translate them with the corresponding senses into a given target language (Marvin and Koehn 2018). When systems fail to select the correct sense and, instead, choose a wrong one, a disambiguation error is produced (see Table 1). According to recent research work, these errors can be related to artifacts located in the training data, which lead to disambiguation biases. These can be described as the tendency of systems to translate a given ambiguous word used with an infrequent meaning into a target language with an incorrect, often predominant, sense (Emelin, Titov, and Sennrich 2020; Campolungo et al. 2022). In this scenario, investigating the disambiguation capabilities of systems involves studying the quantity and nature of semantic errors produced and, consequently, detecting the presence of disambiguation biases. However, despite the work proposed so far, we still lack a comprehensive study of this phenomenon and its impact on automatic translations across different languages. To fill this gap, the present article aims to provide insightful answers to the following research questions:

• RQ1 What is the impact of lexical ambiguity on state-of-the-art MT models and LLMs?

• RQ2 What error patterns can be identified, when analyzing the errors produced by models in translating ambiguous words?

• RQ3 Given an encoder-decoder architecture trained for MT, to what extent does the encoder contribute to distinguishing word senses? Does the encoder learn representations suitable for disambiguating ambiguous words?

• RQ4 What is the relation between the capacity of an architecture and its ability to represent different senses of ambiguous words?

• RQ5 Do multilingual models sacrifice their disambiguation capabilities so as to be able to handle multiple languages?

• RQ6 What is the impact of the beam search decoding strategy on the disambiguation capabilities of MT models?

• RQ7 How effective are standard evaluation settings in assessing the disambiguation ability of MT systems?

This article extends Campolungo et al. (2022), who proposed a manually curated benchmark for investigating disambiguation biases in five language pairs. In our work, we address the aforementioned research questions by bringing about the following novel contributions:

• We extend the benchmark from five to eight languages: Bulgarian, Chinese, Dutch, German, Italian, Russian, Slovene, and Spanish. These languages cover four subgroupings of the Indo-European and Sino-Tibetan language families: Balto-Slavic, Germanic, Italic, and Sinitic.

• We significantly increase the coverage of the benchmark by manually expanding and refining the sets of good and bad translations.

• Compared with Campolungo et al. (2022) who tested only seven MT systems, we include four additional state-of-the-art MT systems and eight LLMs.

• We carry out a novel manual error analysis to identify and describe the error patterns shown by the tested systems when dealing with lexical ambiguity.

• We propose an extensive array of experiments:

  • (i)

    We investigate the impact of MT systems’ encoder module on their disambiguation capabilities, thereby studying the effectiveness of the representations learned by the encoder for Word Sense Disambiguation (WSD).

  • (ii)

    We inspect the extent to which the capacity of a model is related to its WSD capabilities, and whether multilingual MT systems sacrifice their ability to disambiguate rarer senses in order to be able to deal with multiple languages.

  • (iii)

    We study the role of the beam search algorithm in dealing with lexical ambiguity, and whether systems are inherently biased toward (more common) senses of ambiguous words.

  • (iv)

    We explore the suitability of standard MT evaluation settings for detecting disambiguation errors, thus highlighting the need for a manually curated benchmark such as DiBiMT.

One of the first contributions aimed at assessing the ability of MT models to handle lexical ambiguity is proposed by Mihalcea, Sinha, and McCarthy (2010), who introduce the Cross-Lingual Lexical Substitution (clls) task. Inspired by McCarthy and Navigli (2007), who put forward the English Lexical Substitution task at SemEval-2007, clls requires systems to provide a substitute for a focus word in context in a given target language. For example, a candidate system for English-to-Italian cross-lingual lexical substitution provided with the sentence He removed the top of the carton would be expected to suggest the substitutes coperchio and parte superiore for the focus word top. Instead, if the system is provided with the sentence Put a top on the toothpaste tube, such system would be expected to propose the word tappo as a substitute for top. Although the focus word is the same in both sentences, the two substitutes differ. Crucially, clls does not rely on any sense inventory, which presents a significant limitation when it comes to assessing the disambiguation capabilities of MT models. In fact, since focus words, which can be ambiguous, are not associated with a semantic tag, it is not possible to determine their meaning. As a result, the nature of potential semantic errors and biases cannot be investigated effectively. Furthermore, the task focuses solely on one language pair, namely, English-Spanish, and provides only a development set composed of 30 words and a test set of 100 words.

The aforementioned limitations are partially addressed by Lefever and Hoste (2010 , 2013) who propose the Cross-Lingual Word Sense Disambiguation (clwsd) task. Here, systems are required to choose the most appropriate translations from a set of possible candidates, for a given word in context. For instance, a system provided with the sentence He removed the top of the carton and candidate translations for top, that is, coperchio, tappo, and parte superiore, among others, should select only coperchio and parte superiore as correct translations. This approach addresses the well-known drawbacks of traditional WSD (Navigli 2009; Bevilacqua et al. 2021), such as the fine granularity and static nature of sense distinctions, and even the need for a wide coverage sense inventory. However, while some characteristics of the aforementioned task formulation are advantageous from a WSD standpoint, these might prove detrimental to the detection of semantic biases in MT. In fact, similarly to the clwsd task, the lack of wide-coverage sense inventories hampers the study of the nature of disambiguation biases, making it impossible to determine whether a given incorrect sense chosen by an MT model is more frequent than the corresponding correct one. Moreover, the test dataset focuses on 20 ambiguous focus words, covering only one part of speech, namely, nouns. Finally, clwsd relies on an automatically built sense inventory to select translation candidates. This inventory is derived from Europarl (Koehn 2005), which mostly covers the political domain; the test sentences are drawn from the American National Corpus (Ide and Suderman 2004), which, instead, encompasses different topics, thus introducing potential coverage issues in the sense inventory due to the topic mismatch.

Focused on investigating the disambiguation capabilities of MT systems, Gonzales, Mascarell, and Sennrich (2017, ContraWSD) propose a new contrastive evaluation dataset, in which every instance is composed of three elements: (i) a source sentence s containing a focus word; (ii) its reference translation t, and (iii) a set of contrastive examples C = {c₁,…, c_n}, where the translation for the focus word is replaced with the translation of one of its other meanings. For each instance, a candidate MT system is required to assign a probability score to the pairs composed of s and t as well as s and all other contrastive sentences in C. Given a function ρ which assigns a probability score to a given (source, translation) pair, an instance is considered correctly classified if ρ(t|s) > ρ(c_i|s)∀c_i ∈ C, in other words, the pair containing the reference translation is assigned a higher score. ContraWSD includes 7,200 instances for German → English, and 6,700 for German → French. While the contrastive formulation simplifies the process of evaluating models by requiring only a scoring function, such a formulation exhibits a number of significant drawbacks. First, this evaluation disregards the fact that a given MT system might never generate either the translation labeled as correct or the contrastive ones, which exposes the evaluation to the degree to which a given translation is in-distribution compared to the training data. Moreover, while the authors enforce that the correct and incorrect translations agree in number and gender (depending on the target language), models could assign lower scores due to poor fluency of the artificially created sentences, rather than actual semantic errors, thanks to the strong language modeling capabilities exhibited by MT models (Voita, Sennrich, and Titov 2021).

Building upon this work, Rios, Müller, and Sennrich (2018) present the Word Sense Disambiguation Test Suite, that is, a denoised version of ContraWSD focused on the German → English language combination. Here, rather than scoring translations, the WSD Test Suite evaluates the MT output directly. Importantly, this new dataset introduces a crucial constraint that enables straightforward disambiguation of the sense chosen by candidate models, by including only sentences in which the translation of the ambiguous source word cannot refer to different meanings. In order to illustrate this constraint, let us consider the example provided by the authors: The German ambiguous word Stelle can refer to two different senses, namely, job and place, intended as a paid position of regular employment and as a particular position, point, or area in space; a location, respectively; however, since both of these senses could be translated into English with position, the authors remove the word Stelle from the dataset. WSD Test Suite represents a significant leap toward a reliable evaluation of semantic biases in automatic translations, with 3,249 sentence pairs, each targeting one of the 20 ambiguous German words in the dataset, and totaling 45 distinct word senses. Despite the aforementioned improvements, the WSD Test Suite displays limited coverage in terms of words and senses, and features only one translation direction, namely from English to German, while still exhibiting the significant drawbacks that the contrastive formulation suffers from, as illustrated above.

The first effort to create a large-scale contrastive dataset for detecting disambiguation biases is made by Raganato, Scherrer, and Tiedemann (2019), who propose the Multilingual Contrastive WSD benchmark (MuCoW). Inspired by both ContraWSD and the WSD Test Suite, MuCoW is an automatically created test suite available in two variants, namely, the scoring and the translation variant. While the former contains more than 200,000 sentence pairs derived from word-aligned parallel corpora in 16 translation directions, the latter includes 9 language pairs with a total of 15,600 sentences. The MuCoW dataset is created in three steps. First, an alignment tool is used to obtain aligned word pairs occurring more than 10 times in parallel corpora from the OPUS collection (Tiedemann 2012), each connected to at least two distinct focus words. Second, the authors cluster the translations by relying on BabelNet (Navigli and Ponzetto 2010; Navigli et al. 2021) and sense embedding similarity, so as to address BabelNet’s fine granularity. Third, random sentences are selected for a given word pair and the target word is replaced with a lexicalization pertaining to a synset other than that associated with the focus ambiguous word, thus constructing several contrastive instances for a given pair (scoring variant only). Despite the aforementioned advantages, we highlight two key limitations in MuCoW over and beyond the contrastive formulation itself, whose drawbacks have been illustrated above. First, due to its entirely automatic construction, MuCoW is prone to contain errors and noise derived from the underlying semi-automatic sense inventory. Second, such a benchmark relies on the problematic assumption that the synonyms associated with a specific synset can be used interchangeably for translating a given focus word.

Moving in a different direction from previous works, Emelin, Titov, and Sennrich (2020) take a step toward model analysis, thoroughly exploring the correlation between biases picked up by MT models and the distribution of their training data. Interestingly, the authors leverage this correlation to create two challenge sets: (i) a WSD bias challenge set, built to quantify the intrinsic bias caused by an ambiguous word’s context; and (ii) an adversarial challenge set, built to measure models’ susceptibility to adversarial injection of terms usually associated with meanings other than the target one. Both challenge sets rely on manually refined sense clusters, initially built by automatically merging together BabelNet synsets. Each sense cluster contains an ambiguous English word and a set of monosemous German words, such that, considered jointly, they uniquely identify a specific meaning. Crucially, this work requires the availability of training data which are not always accessible. Furthermore, it focuses on just one language combination, namely, English → German. The approach relies solely on the accuracy score to evaluate models, and this can only provide partial information regarding the biases exhibited by models.

Based on the findings, drawbacks, and open research questions discussed in previous works, Campolungo et al. (2022) propose DiBiMT as a framework aimed at investigating not only the presence but also the nature and properties of semantic biases in MT in multiple language combinations, covering both nominal and verbal senses. In this article, we consolidate and extend the aforementioned framework and put forward a thorough study of the impact of lexical ambiguity on MT.

We now introduce Disambiguation Biases in MT (DiBiMT), an entirely manually curated benchmark aimed at studying the ability of MT systems to choose the correct sense when translating an ambiguous source word occurring in a given context. Our benchmark requires a model to translate a sentence containing an ambiguous source word and evaluates the correctness of its translation.

We first illustrate the composition of the benchmark as well as its creation process consisting of both automatic and manual steps. Subsequently, we detail the evaluation procedure and the metrics used in the DiBiMT benchmark.

Each instance in the benchmark is defined as a tuple $i=(s,w,σ,G,B)$ composed of the following elements:

• i.

  one source sentence s containing an ambiguous source word w which is associated with a given meaning σ. In our benchmark, due to its predominance in MT, we pivot on the English language and use it as the reference language for all our source sentences. The meaning σ corresponds to a synset in BabelNet⁴ (Navigli and Ponzetto 2010; Navigli et al. 2021), the largest multilingual encyclopedic dictionary which follows and extends the WordNet synset model, where a given concept is represented through a set of lexicalizations which are used in different languages to express such concept.

• ii.

  a set of good translations⁵$G$ for the ambiguous word w in the context of s.

• iii.

  a set of bad translations⁶ℬ for the ambiguous word w in the context of s.

We note that the ambiguous source word w, and both GOOD and BAD translations, can be multi-word expressions and compounds. Furthermore, translations can have a different part of speech than the corresponding source word. As target languages, that is, the languages of the GOOD and BAD translations for each source word, DiBiMT covers: Bulgarian, Chinese, Dutch, German, Italian, Russian, Slovene, and Spanish. Figure 1 illustrates examples of GOOD and BAD translations for the instance (he poured a shot of whiskey, shot, bn:00057755n⁷, {trago_ES, chupito_ES, goccio_IT, …, glaasje_NL}, {Schlag_DE, sparo_IT, prik_NL, …, выстрел_RU}).

A candidate instance $i=(s,w,σ,G,B)$ of our benchmark is extracted from two lexical-semantic resources, namely, WordNet (Miller 1995) and the English Wiktionary.⁸ As far as WordNet is concerned, we consider each sentence s from the Princeton WordNet Gloss Corpus (Langone, Haskell, and Miller 2004) that is a usage example of a word w manually tagged with a specific WordNet synset. For Wiktionary, we extract every usage example s of a word w, excluding archaic usages and slang. We use the linkage between WordNet synsets (or Wiktionary definitions) and BabelNet synsets to provide a synset annotation σ for our instance i.

We are now left with the task of populating the set of GOOD and BAD translations $G$ and ℬ for each instance i in all eight target languages. Given the BabelNet synset σ in instance i, we define Λ_L(σ) as the set of lexicalizations of σ in language L contained within synset σ. For instance, let us consider the synset $σ~$ of duck defined as broad-billed swimming bird. $σ~$ contains lexicalizations in different languages such as: Ente in German, anatra in Italian, ymкa in Russian, and pato in Spanish. Hence, $Λen(σ~)=${duck}, while $Λit(σ~)={anatra,anitra}$⁠. We then define $GL$ and ℬ_L as the set of GOOD and BAD translations in the target language L. We pre-populate these two sets automatically as follows. We assign $GL=ΛL(σ)$⁠, that is, the set of senses in language L derived from the BabelNet synset σ. Instead, we consider as BAD translations all the lexicalizations in language L of any other synset containing w, except those in $GL$⁠.

For example, consider an instance with w = bank and σ defined as a flight maneuver. Λ_IT(σ) includes the lexicalization avvitamento, which is ambiguous in Italian, as it can refer either to the laying of a screw or a downward spiral (e.g., the downward spiral of costs), among other meanings. Therefore, to satisfy our third requirement, we ensure that no other possible senses of avvitamento (i.e., found in a different synset σ′) can be translated with bank into English according to BabelNet.

As a result of the aforementioned automatic process, we extract 948 candidate sentences. We now proceed to manually validate the entire dataset resulting from the previous procedure. To this end, we employ six annotators with proven experience in lexical semantics and MT.⁹

Annotators are asked to validate both $GL$ and ℬ_L, which contain the set of GOOD and BAD translations for the ambiguous source word w, respectively. Concerning GOOD translations, annotators are asked not only to validate the candidates proposed automatically, but also to include additional translations. Instead, regarding BAD translations, annotators are only required to establish the incorrectness of the automatically extracted translations. To determine the correctness of a candidate translation for w, annotators are required to evaluate its suitability to the context provided, regardless of whether this translation pertains to a part of speech other than that of w. Furthermore, annotators are instructed to analyze the automatically collected sentences as well as the ambiguous source words and verify compliance with the requirements, among which we highlight that: (i) each sentence must provide a semantic context such that the meaning of the ambiguous source word w can be determined unequivocally; (ii) w cannot be part of a complex lexical unit such as an idiomatic expression; and (iii) the sentence is complete (e.g., not just a phrase). We report the complete annotation guidelines adopted in Appendix A.

As a result of the aforementioned manual process, approximately 30% of the 948 instances are discarded due to non-compliance with the guidelines, resulting in 667 instances. As shown in Table 2, on average, 66.4% of the correct translations are added manually by the annotators, whereas only 33.6% are derived automatically from BabelNet. Moreover, on average, almost 60% of the synset lexicalizations in a given language cannot be considered as GOOD translations for the corresponding ambiguous word in each instance. On the one hand, these statistics confirm that synonyms cannot be used interchangeably in translation. On the other hand, they are evidence of the noise contained in automatic lexical semantic resources such as BabelNet. This suggests that high quality in benchmarks such as DiBiMT cannot be achieved by merely relying on automatic resources and without manual intervention.

The summary statistics of the annotated dataset are reported in Table 3. DiBiMT covers 525 different synsets and 358 English lemmas, constituting a total of 667 manually curated instances. The benchmark is available in eight language combinations, that is, from English to each of the following languages: Bulgarian, Chinese, Dutch, German, Italian, Russian, Slovene, and Spanish. Furthermore, we compute the distribution of BabelNet domains over the synsets included in our instance set. As shown in Figure 2, DiBiMT is a heterogeneous benchmark, covering a wide range of domains.

We now describe the automatic procedure that we adopt to determine the correctness of a translation provided by a candidate MT model according to DiBiMT. Given a DiBiMT instance i and a translation τ of sentence s produced by a candidate model into language L, we use Stanza (Qi et al. 2020) to perform tokenization, part-of-speech tagging, and lemmatization of τ. The analysis procedure computes the lexical overlap between the set of lemmas in τ and the two sets of manually validated translations, that is, $GL$ and ℬ_L. When dealing with multi-word expressions in GOOD or BAD translations, annotators are allowed to use wildcards so as to enable matching of non-contiguous spans (e.g., auf * Wünsche eingehen which means to attend to somebody’s wishes, where * represents any sequence of words). Additionally, since Stanza features multi-word expansion tokenization for some of the languages considered, whenever possible, we perform matching on both the list of words and tokens in the translated sentence.

Finally, we label a given instance as: (i) MISS if no match is found; (ii) UNK in the case that both GOOD and BAD translation sets match; and (iii) GOOD or BAD depending on which set matches a given lemma contained in the translation produced by the MT model.

By manually inspecting misclassified instances, we notice that one of the main causes is to be found in textual pre-processing steps such as tokenization, lemmatization, and part-of-speech tagging. In order to minimize pre-processing errors and reduce MISS tags, we apply multiple Stanza models whenever available for the same language,¹⁰ owing to the fact that each model is trained on a specific treebank from Universal Dependencies (Nivre et al. 2016), and each treebank adopts its own domain and annotation guidelines. Specifically, we use the following packages: GSD and HDT for German; AnCora and GSD for Spanish; Combined,¹¹ ISDT, ParTUT, VIT, TWITTIRO, and PoSTWITA for Italian; SyntagRus, GSD, and Taiga for Russian; GSDSimp¹² for Chinese; BTB for Bulgarian; Alpino and LassySmall for Dutch; and SSJ, SST, and, additionally, CLASSLA for Slovene (Ljubešić and Dobrovoljc 2019).

In addition to the aforementioned improvements, we enrich our analysis procedure with auxiliary heuristics dealing with specific linguistic phenomena, such as compounds and reflexive verbs, and aimed at further reducing the MISS instances.

Preliminary analyses show that the standard evaluation procedure fails to recognize GOOD or BAD translations represented by reflexive verbs in languages such as German, Italian, and Spanish. For the latter two languages, we address these cases as follows. First, we identify reflexive verbs included as GOOD and BAD translations by inspecting the ending, that is, rsi and rse in Italian and Spanish, respectively. Subsequently, we associate such reflexive verbs with some reflexive pronouns in the corresponding language, that is, mi, ti, si, ci, vi in Italian and me, te, se, nos, os in Spanish. Finally, we determine whether the lemmatized sentence produced by a given model contains the reflexive form included as GOOD or BAD or the corresponding non-reflexive base form of the verb and one of the associated reflexive pronouns.

Instead, in German we check whether a given GOOD or BAD translation is composed of at least two elements, the first of which is the reflexive pronoun sich. If this is the case, we look for a match between the base form of the verb that comes after sich and any of the reflexive pronouns mich, dich, sich, uns, euch in the lemmatized candidate translation. We leave it to future work to address multiple cases and scenarios as well as to extend and improve these rules to all languages covered.

The correct detection of a GOOD or BAD translation into German requires additional rules regarding specific linguistic phenomena such as word order, multi-word expressions, compounds (Komposita), and separable verbs. For instance, given the source sentence He thought Nehru jackets went out of fashion at the end of the seventies translated into German with Er dachte, Nehru-Jacken kamen Ende der siebziger Jahre aus der Mode, the correct lemmatization of kamen [...] aus der Mode is aus der Mode kommen. In order to ensure that the GOOD translation aus der Mode kommen is detected in the aforementioned translation, we implement a multi-word inversion mechanism capable of dealing with these cases.

Another challenge when translating into German concerns separable verbs, such as zurücklegen. For example, given the source text The caravan covered almost 100 miles each day translated with Die Karawane legte jeden Tag fast 100 Meilen zurück, the correct lemmatization of legte [...] zurück is zurücklegen. To address these specific cases, we rely on the dependency tree, which classifies the nature of the relationship between prefixes of separable verbs and the main part of the verb using the Universal Dependencies¹³ dependency relation compound:prt. Specifically, we combine the prefix with the lemmatized core part of the verb, obtaining forms such as zurücklegen.

We observe that, in some cases, instances are classified both as GOOD and BAD; we define these cases as UNK instances. Interestingly, our manual inspection shows that UNK instances are often caused by the match between a single word w appearing in ℬ and a compound word, or multi-word, containing w, found within $G$⁠. We address these cases by attributing higher priority to the identification of the longest matches, thereby ignoring potential shorter matches contained in longer matches already classified as either GOOD or BAD. Importantly, we note that UNK instances represent, on average, 1.15% of all DiBiMT items, thus having a limited impact on the overall evaluation of semantic biases. We report the breakdown of UNK instances in Appendix D.1.

In this section, we detail the metrics used in the DiBiMT benchmark to evaluate MT models. In addition to accuracy, DiBiMT analyzes the semantic biases of a translation model via four metrics, which we illustrate in what follows. To formally define these metrics, we adopt the following notation:

• λ_P represents a (lemma, part of speech) pair, where P is the part of speech;

• Ω_L(λ_P) = {σ₁,…, σ_n} indicates the set of synsets pertaining to the part of speech P which contain λ as a lexicalization in language L;

• $μλP(σ)$ refers to the index of synset σ in Ω_EN(λ_P) ordered according to WordNet’s sense frequency, as computed from SemCor. That is, index k means that synset σ is the k-th most frequent meaning for λ_P.

All the metrics defined in this section are computed on the set of translations produced by a given model ℳ in a target language L, excluding MISS and UNK occurrences.

The standard accuracy metric provides a general view of a model’s performance in a given language. It is computed as: $#GOOD#GOOD+#BAD$⁠.

This metric assesses the impact of frequency on MT models’ ability to handle ambiguous words. First, we group instances based on their ambiguous word meaning’s frequency index, that is, $μλP(σ)$⁠. We define #GOOD_μ as the number of GOOD instances whose ambiguous word’s meaning has a frequency index of μ; similarly, #BAD_μ is defined for BAD instances. Moreover, we introduce the set of all frequency indices a synset can assume in the DiBiMT dataset as $M={μλP(σ)∣λP∈$ LP, σ is the meaning of λ_P in context }, where LP represents the set of all (lemma, part of speech) pairs corresponding to the target words in DiBiMT.

As a result, instances focused on lower-frequency senses are given more prominence in the SFI metric. We discuss our choice of using frequency index-based weighting instead of frequency-based weighting in Appendix B.

This metric investigates the impact of the polysemy degree of words on the disambiguation capabilities of a model. To this end, we define δ_L(λ_P) = |Ω_L(λ_P)| as the polysemy degree, that is, the number of senses of λ_P in language L. This metric mirrors SFI in the way it is computed, but groups instances by their ambiguous word’s polysemy degree δ_EN(λ_P) instead of μ.

Finally, we measure the frequency with which models predict senses that are more common than the correct one. Given an instance classified as BAD, we denote $σ^$ as the synset associated with the incorrectly translated lemma.¹⁴ Subsequently, we compute the frequency index of σ and $σ^$ with respect to λ_P. If $μλP(σ^)<μλP(σ)$⁠, then the system selected a sense which is more frequent than the correct one and which we call More Frequent Sense (MFS+). Additionally, if $μλP(σ^)=1$⁠, then the model disambiguated the source word w to the Most Frequent Sense (MFS) of the associated lemma λ_P.

We now put our DiBiMT framework into practice, analyzing and discussing the performance obtained by state-of-the-art MT systems and LLMs. First, we describe the systems considered. Then, we detail critical issues that surfaced as a result of preliminary experiments as well as our strategies to address them. Lastly, we illustrate our results.

We evaluate a wide range of systems, including commercial and non-commercial MT models and LLMs. Specifically, we test the following MT systems:

• Google Translate,¹⁵ one of the most popular commercial MT systems.

• DeepL Translator,¹⁶ a state-of-the-art commercial MT system.

• MBart50 (Tang et al. 2021), multilingual BART fine-tuned on the translation task for 50 languages (around 610M parameters). We refer to MBart50 as the English-to-many model, and to MBart50_mtm as the many-to-many model.

• M2M100 (Fan et al. 2021), a multilingual model able to translate from/to 100 languages. We evaluate both versions of the model, the one with 418M parameters (M2M100) and the one featuring 1.2B parameters (M2M100_lg).

• OPUS (Tiedemann and Thottingal 2020), one of the smallest state-of-the-art NMT models available to date; a base Transformer (each model features approximately 74M parameters) trained on a single language pair on large amounts of data. Moreover, we evaluate its multilingual counterpart, jointly trained on multiple target languages, which we dub OPUS_mul.¹⁷

• NLLB-200 (NLLB Team et al. 2022), a state-of-the-art MT system trained on approximately 200 different languages. NLLB-200 leverages multiple data augmentation techniques, such as back-translation via both neural and statistical MT models, and high-quality bitext mining. Specifically, we consider the following four different versions of NLLB-200:

  • –

    NLLB-200_sm: smallest version, distilled, counting 600M parameters;

  • –

    NLLB-200_md: medium version, counting 1.3B parameters;

  • –

    NLLB-200_mdd: medium version, distilled, counting 1.3B parameters;

  • –

    NLLB-200_lg: large version, the baseline model presented in NLLB Team et al. (2022), counting 3.3B parameters.

Furthermore, we evaluate the following LLMs:

• Gemma 1 (Gemma Team et al. 2024a), a family of open-source language models based on Google’s Gemini (Gemini Team et al. 2024). Gemma is based on the Transformer decoder architecture (Vaswani et al. 2017), relying on crucial improvements such as the MultiQuery Attention (Shazeer 2019), rotary positional embeddings (Su et al. 2024), and the GeGLU activation function (Shazeer 2020). We use:

  • –

    Gemma-2B (google/gemma-2b-it), which features 2B parameters and is trained on 3T tokens;

  • –

    Gemma-7B (google/gemma-7b-it), which is a 7B model trained on 6T tokens.

• Gemma 2, that includes new additional Gemma models which range in scale from 2B to 27B parameters. We use Gemma2-9B (google/gemma-2-9b-it), with 9B parameters and trained on 8T tokens (Gemma Team et al. 2024b). Compared to Gemma 1 models, Gemma2-9B additionally utilizes the interleaving local-global attentions (Beltagy, Peters, and Cohan 2020) and the Grouped-Query Attention (Ainslie et al. 2023) and is trained using knowledge distillation by minimizing the negative log-likelihood between the probabilities of a larger model, the teacher, and a smaller one, the student.

• LLaMA 2, a family of models developed by Meta (Touvron et al. 2023). Trained on publicly available datasets only, LLaMA 2 models range from 7 to 65 billion parameters. LLaMA 2 models rely on the Transformer architecture, with improvements such as the SwiGLU activation function (Shazeer 2020) and rotary embeddings. We use LLaMA2-7B (meta-llama/Llama-2-7b-chat-hf), a 7 billion parameter conversational model optimized for generating human-like responses in a dialogue setting.

• LLaMA 3, an improved version of the LlaMA 2 model family. LLaMA 3 models are available in two sizes: 8 billion and 70 billion parameters. One notable enhancement in LLaMA 3 is the implementation of a new tokenizer, which significantly increases the vocabulary size to 128,256 tokens, compared with the 32,000 tokens used in LLaMA 2. We evaluate meta-llama/Meta-Llama-3-8B-Instruct, which we dub LLaMA3-8B.

• Mistral-7B, a pretrained LLM featuring 7 billion parameters (Jiang et al. 2023). Its vocabulary size is 32,000. We evaluate mistralai/Mistral-7B-Instruct-v0.2.

• Tower-7B, a 7 billion parameter model fine-tuned for translation-related tasks, such as MT, automatic post-editing, and Named Entity Recognition (Alves et al. 2024). We experiment with Unbabel/TowerInstruct-7B-v0.1. Built on top of LLaMA 2, this model has been fine-tuned to translate between English and the following languages: Chinese, Dutch, French, German, Italian, Korean, Portuguese, Russian, and Spanish.

• Phi-3, a family of small language models by Microsoft (Abdin et al. 2024). Phi-3 language models are available in both short- and long-context lengths. We evaluate microsoft/Phi-3-mini-128k-instruct, a lightweight language model with 3.8B parameters, supporting a context length of 128K tokens. microsoft/Phi-3-mini-128k-instruct is trained on 3.3T tokens and its vocabulary size is 32,064.

• GPT-3.5_turbo, a large language model optimized for chat applications. The translations were produced by the default model at the time of writing, that is, gpt-3.5-turbo-0613.

• GPT-4, a large-scale multimodal language model that achieves better performance than its competitors (OpenAI et al. 2024). The translations from GPT-4 were produced by the default model at the time of writing, that is, gpt-4-0613.¹⁸

We report in Appendix C the details about the generation parameters we use for each open-source MT system, together with the prompt provided to the LLMs.

To test the aforementioned systems, we adopt the evaluation procedure illustrated in Section 3.3, using the metrics we defined in Section 3.4. Thus, each system is assigned specific scores to account for different aspects of the evaluation, that is, its overall accuracy, its ability to deal with infrequent senses and highly ambiguous words, represented by SFI and PDI metrics, respectively, and the frequency with which it predicts senses that are more common than the correct one, represented by the MFS and MFS+ metrics.

In order to reduce the number of MISS instances we perform a manual refinement, which we illustrate in this section. Specifically, annotators are required to inspect the MISS instances resulting from the evaluation of some of the considered systems and determine whether these should be classified as GOOD or BAD. This step is prompted by the consideration that the set of GOOD and BAD translations is not closed and can always be augmented with new suitable items. We highlight that, due to time and budget constraints, the aforementioned refinement step does not involve the NLLB-200 models, OPUS_mul, and the considered LLMs.

In Table 4 we report the average percentage of MISS instances on the DiBiMT benchmark before and after the MISS refinement, and the related accuracy scores. As we can see, the refinement process substantially reduces the percentage of MISS instances, with an average reduction of 19.75% across all tested systems. Furthermore, we notice that the refinement produces an average accuracy increase of 19.61%, suggesting that a MISS instance might frequently be associated with a correct translation, rather than an incorrect one. From these results, we can also notice that manually inspecting sentences produced by some models does not bias the benchmark toward them. Indeed, the average drop in MISS % in refined and non-refined systems is 22.82% and 18.31%, respectively, and the average accuracy increase is 18.49% and 20.14%, respectively, with very similar deltas between the two groups.

However, we wish to point out two notable exceptions. The first concerns OPUS_mul, for which we report a substantially smaller MISS % drop and accuracy increase, compared with other models. We attribute this to its particularly low accuracy, as will be discussed in the next section. Indeed, systems struggling to translate ambiguous focus words often generate contextually inappropriate tokens or severe hallucinations, which arguably should not be included in the benchmark as they are not disambiguation errors. We report examples of these phenomena in the qualitative analysis of Section 5. The second exception is represented by the open-access LLMs. As we can see from Table 4, these systems display a much higher percentage of MISS instances than the others. We attribute this to the fact that, to the best of our knowledge, they perform MT in a zero-shot setting, without prior fine-tuning for the task.¹⁹ Indeed, it has already been shown that the zero-shot translation performance of most LLMs can be substantially improved by either using few-shot learning (Zhang, Haddow, and Birch 2023; Bawden and Yvon 2023), or by fine-tuning them to MT (Alves et al. 2024; Xu et al. 2024). Also, all the tested LLMs were pre-trained using mostly English data, with very limited exposure to some of the target languages in the DiBiMT benchmark. As a consequence of the foregoing factors, open-access LLMs frequently hallucinate and produce omissions. We delve deeper into this matter in our linguistic analysis, in Section 5.

Finally, we report a total average of 38.14% MISS instances. We wish to highlight that this number is inflated by the high percentage of MISS instances of the open-access LLMs. Indeed, considering only the other systems, the average MISS % is reduced to 31.97%. Additionally, we argue that some translations are ultimately non-classifiable, for example, a complete omission of the target word could still be correct if the translation conveys the meaning of the source sentence, as we show in the linguistic analysis of Section 5. Finally, we believe that manually accounting for any possible translation produced by all existing models is unfeasible, due to the high variability intrinsic to the translation task.

We report the accuracy of the aforementioned systems on the DiBiMT benchmark in Table 5. We observe average values ranging from 20.06% to 71.09%, with an absolute low of 9.76% (OPUS_mul in Chinese), and an absolute high of 83.37% (Gemma2-9B in Russian).

The best-performing system is GPT-4, with an average accuracy of 71.09%, followed by Gemma2-9B and GPT-3.5_turbo. We note that the largest (and more recent) LLMs achieve competitive results despite their not being fine-tuned directly for MT. This confirms the results of Iyer, Chen, and Birch (2023), where the authors found that instruction-tuned LLMs achieve state-of-the-art performance when dealing with disambiguation biases, with competitive scores even in unseen languages.²⁰ Interestingly, Gemma2-9B achieves an accuracy close to that of GPT-4, and better than one of the best commercial translation systems, that is, DeepL, despite having only 9B parameters. Also, Tower-7B, an LLM based on LLaMA2-7B additionally fine-tuned for MT-related tasks, achieves performance that is competitive with the larger LLMs and commercial MT systems.²¹ This suggests that a promising direction for MT research could be eliciting the translation performance of LLMs. Indeed, sense (in)frequency poses one of the hardest challenges for MT systems (Section 4.4.1). Therefore, finding ways to leverage the information LLMs learn from monolingual corpora, which is abundant, might be key to mitigating this issue. In addition, thanks to the vast amounts of monolingual data available, LLMs can be scaled to accommodate higher parameter counts, learning more complex relations between words and their contexts. Indeed, the highest performance was obtained by GPT-4 which, most likely, is the largest system among the ones considered here.²²

Among the open-source MT systems, instead, NLLB-200 variants outperform the others, with the larger NLLB-200_lg being the only open-source system that obtains results that are competitive with the commercial systems. Indeed, although on average commercial systems perform considerably better than their non-commercial counterparts, we observe that NLLB-200_lg manages to outperform Google Translate in every language except Spanish, Russian, and Chinese. The systems that attain the lowest accuracy are M2M100 and OPUS_mul, that is, the smallest ones, together with OPUS_bil. As expected, we observe that, for a given architecture, increasing the parameter count corresponds to an increase in accuracy; for instance, NLLB-200_lg obtains a mean accuracy of 57.82%, while NLLB-200_sm only 40.55% with a decrease of 17.27%, while M2M100_lg outperforms M2M100 by 5.95%. Surprisingly, this does not apply for OPUS_bil, whose overall accuracy is higher than that of OPUS_mul, M2M100, M2M100_lg, MBart50, and MBart50_mtm, despite the last four being considerably larger. We defer the explanation of this phenomenon to Section 6.3.

Concerning the availability of resources in specific language pairs, we note that the mean accuracy for Chinese is particularly low, despite the large number of parallel corpora available for this translation direction. Furthermore, although Fan et al. (2021) consider German to be high-resource, compared with Slovene, M2M100 and M2M100_lg exhibit better performance in Slovene—with accuracy scores of 32.41% and 36.02%—than they do in German, where their scores are 26.25% and 31.84%, respectively. The same pattern is observed in the outputs of OPUS_mul. Finally, we highlight that, while substantial efforts are being made to improve the availability of parallel corpora for low-resource languages, we observe that, despite its low number of MISS instances, a high-resource language such as Dutch exhibits the poorest performance in DiBiMT (see Table 5). In light of this counterintuitive finding, we encourage the MT community to assess and address the impact of lexical ambiguity on high-resource languages as well, as it might still pose significant challenges for these.

In Figure 3, we present the mean accuracy, SFI, and PDI scores for each system. As expected, both SFI and PDI exhibit lower values compared to accuracy, which is due to the systems’ increased difficulty in accurately translating words with rare meanings and those that are highly ambiguous. Interestingly, SFI is consistently lower than PDI, suggesting that the infrequency of senses poses a particularly significant challenge for neural architectures.

To investigate this matter more thoroughly, we group the instances of DiBiMT based on their sense frequency index and polysemy degree, and present in Figure 4 the accuracy of each system on each identified group.²³ As anticipated, the accuracy significantly drops when the focus word exhibits a high polysemy degree, or its associated meaning is infrequent, and the decline in accuracy is more evident with increasing levels of frequency index, rather than polysemy degree. Interestingly, the most accurate systems demonstrate greater robustness to highly polysemous words. To illustrate this, we compare the average accuracy decrease when transitioning from words with a polysemy degree of less than 6 to those with 14 or more. The average drop in accuracy is 22.55 points; however, for top-performing systems (GPT-4, Gemma2-9B, GPT-3.5_turbo, DeepL, Tower-7B, NLLB-200_lg, Google, LLaMA3-8B, NLLB-200_mdd, and NLLB-200_md), this decrease is less marked, at 17.17 points, versus 27.03 points for the other systems. We hypothesize that systems with limited contextualization capabilities struggle when selecting the correct translation from numerous possibilities. Instead, more accurate systems are trained on extensive data and have likely learned better decision boundaries for different senses of ambiguous words, making them more confident in picking translations representing the correct meaning, regardless of the number of possible translations.

Conversely, all systems seem to encounter similar difficulties when ambiguous words are used with particularly infrequent meanings, that are likely underrepresented in their training corpora. Indeed, the average accuracy drop between the most frequent sense (<2) and senses over the 9th frequency index is 34.51, and the gap between the average score of top- and bottom-performing systems is only 5.06, much lower than the gap we observe for PDI, at 9.86. In these cases, all systems tend to translate the focus word into one of its more common meanings. We delve deeper into this phenomenon in Appendix D, where Table 19 presents the scores of MFS and MFS+ metrics, which quantify how often a system translates the focus word into its most frequent sense (MFS), or a sense more frequent than the correct one (MFS+). On average, we report that the MFS is chosen more than half of the time a model translates incorrectly, whereas systems choose a more frequent sense more than 3 out of 4 times they make a mistake. We hypothesize that this tendency to translate into more frequent meanings is strictly related to the balance of senses in the training corpora. Indeed, since words’ meanings appear in natural texts following a Zipfian distribution, neural models may tend to fall back upon this distribution whenever they are unsure about the meaning of an ambiguous word. In Section 4.4, we suggested exploring ways for leveraging monolingual corpora to counter the sense infrequency issue. In addition, we believe that another promising direction for MT research might be re-balancing the sense distribution found in monolingual and parallel corpora, to incorporate a larger number of occurrences of words used with infrequent meanings. However, we encourage caution and suggest using the DiBiMT benchmark to measure variations in the disambiguation accuracy of words with frequent and infrequent meanings. Indeed, modifying the sense distribution of MT systems’ training data might have the undesirable outcome of trading off the disambiguation performance of frequent senses for infrequent ones.

Notably, verbs are found to be harder to disambiguate than nouns (Barba, Procopio, and Navigli 2021), also considering the average polysemy degree of nouns and verbs in WordNet, namely, 1.24 and 2.17, respectively. We investigate whether this is true when evaluating systems against the DiBiMT benchmark. With this aim in view, Table 6 shows the average performance considered both on all parts of speech in DiBiMT, and on nouns and verbs separately. While the accuracy scores remain similar, we observe a significantly higher number of MISS instances for verbs. Interestingly, on average, the only notable difference seems to be that systems are more biased toward the most frequent sense when dealing with nouns. We attribute this to verbs having a higher average polysemy degree, and therefore a higher number of more frequent senses, making it more likely for systems to pick one or the other of these rather than homing in on the single most frequent one.

In this section, we address RQ2, that is, we investigate the presence of patterns when analyzing the errors produced by the systems under consideration when translating ambiguous focus words. As a result of our analysis, we identify the following four error patterns: (i) disambiguation errors, in which models fail to choose the correct sense, and, instead, select an incorrect sense, which is often more frequent than the correct one; (ii) omissions, when systems do not translate the focus word; (iii) untranslated source words, that is, the focus word is reproduced as-is in the translation; and (iv) hallucinations, when the focus word is translated with a word or expression that is semantically detached from the source, including when models generate a word or expression that does not exist in the target language. In the following, we introduce and discuss these error patterns. Finally, we focus on errors encountered in general-purpose LLMs.

For the purposes of investigating the ability of MT systems to deal with lexical ambiguity, the most relevant error pattern to be studied is represented by disambiguation errors (also referred to as WSD errors), that is, translation errors in which systems do not select the correct sense for the ambiguous focus word.

In this error scenario, the incorrect sense chosen by a given system corresponds either to the most frequent meaning denoted by the focus word or simply a meaning that is more frequent than the correct one. For instance, the source text The pitcher delivered the ball is incorrectly translated into Dutch by Google with De werper leverde de bal af, in which the verb afleveren does not convey the correct meaning of the focus word, that is, throw or hurl from the mound to the batter, as in baseball, which, instead can be expressed by the Dutch verb gooien. Similarly, the focus word call in The ship will call in Honolulu tomorrow, used with the meaning stop at a given station on a specific route, is incorrectly translated by many models into different languages with more frequent senses of the verb to call (e.g., to assign a name or to get into communication).

Interestingly, we observe disambiguation errors when the focus word is used with a figurative meaning. For instance, in the source text The continuous rain washed out the cricket match, the focus word wash out means to prevent or interrupt due to rain. The aforementioned source sentence is translated by OPUS_bil into Bulgarian with Дъждътотмимача за крикет, which corresponds to The rain washed away the cricket game. Similarly, the sentence The storm had washed out the game is rendered by DeepL with Бурята еотмилаиграта, in which the proposed translation for the focus word once again means wash away.

Moreover, we identify specific source texts in which the focus word is incorrectly translated into all languages by the majority of systems. For instance, in the sentence She was checking out the apples that the customer had put on the conveyer belt, the focus word check out is translated by several models with verbs meaning checking or examining the quality or accuracy of something and not record, add up, and receive payment for items purchased. A potential explanation for this phenomenon could be the fact that the correct senses of the focus words above are much less frequently encountered—if at all—in the training data than the ones chosen by systems. Should this hypothesis be verified, then counterbalancing the senses in the training data might prove beneficial to improving the disambiguation capabilities of systems. Additional instances of disambiguation errors are reported in Table 7.

This type of error consists in omitting the translation of the ambiguous focus word. Specifically, we identify two types of omission, that is, severe omissions and mild omissions, depending on the impact of the omission on the quality and comprehensibility of the output translation. Instances of the two types of omission are reported in Tables 8 and 9, respectively.

Severe omissions compromise the translation system output. This type of omission can affect not only the focus word, but also textual segments in which the focus word occurs and, in some rarer cases, even entire clauses. For instance, considering Dutch as the target language, we observe that, in the source text I don’t want to be bald, so just top my hair, the coordinate clause so just top my hair is omitted entirely, as reported in Table 8.

An instance of single focus word omission is represented by the translation into German and Dutch of the source text Bacon is very fatty when raw; however, most of the fat will render during cooking. Here, MBart50 omits the focus word render which describes the action carried out by the subject in the second clause and whose omission impairs the comprehensibility and grammaticality of the output text severely, as shown in Table 8. Another instance of omission present in several target languages can be identified when considering the focus word in the source text Beg the point in the discussion which is omitted by several systems including NLLB-200_md.

Interestingly, we observe instances that, unlike the previous example, preserve the grammatical correctness of the output translation despite the presence of a severe omission. For instance, M2M100 translates the source sentence We drew last time we played into Italian with L’ultima volta abbiamo giocato (meaning The last time we played). Here, while the output text shows overall grammatical correctness, the omission of the verb drew leads to the removal of crucial information contained in the source text, namely, that of tying a game. Along these lines, the source sentence The dog’s laps were warm and wet is translated into many languages with the rather bizarre equivalents of either The dog is warm and wet or The dogs are warm and wet (Table 8).

Finally, a noteworthy case of severe omission across models and languages can be observed when considering the source text she made gravy with a base of beef stock, whose translations show that several models omit the focus word stock.

While severe omissions compromise the quality of the translation significantly, mild omissions exhibit a minor detrimental impact on the intelligibility of the output translation and its adherence to the source text. An interesting instance of this scenario can be observed in Bulgarian, where non-numeral quantifiers such as fix in its meaning a dose of something strongly desired can be omitted with a slight loss of information but without failing to convey the overall meaning of the translation. For example, the sentence She needed a fix of chocolate can be rendered, omitting the focus word fix, with the equivalent of She needed chocolate, that is, in Bulgarian with Тя имаше нужда от шоколад, as proposed by both DeepL and Google. Interestingly, we also observe this type of omission in other target languages such as Slovene. For instance, phrases like roll of thunder and clap of thunder are rendered with the equivalent of thunder or to thunder by DeepL and Google, specifically with grmenje and zagrmeti, respectively. Finally, an additional representative example of this phenomenon can be observed when considering the translation into Slovene of the source text The valley was between two ranges of hills, where the textual sequence between two ranges of hills is translated by DeepL and Google with the equivalent of between two hills (Table 9), that is, omitting ranges. This pattern is the least problematic among those found in our analysis and such translations can be considered correct in some cases, yet this raises interesting questions as to the extent to which a translation should be faithful to the source sentence while maintaining the overall meaning.

Another error pattern identified during our qualitative analysis is represented by untranslated source words, that is, focus words reported as-is in the translation and therefore not translated into the target language. Importantly, we highlight that this error category does not include those cases in which a given source word in English can be considered as an acceptable translation into the target language, for example, English words commonly used in several other languages such as hobby or hotel, or, similarly, anglicisms used in specific domains such as computing.

Interestingly, we observe this type of error when a given focus word has come into use in a specific target language. As reported in Table 10, the source sentence You’d better back up these files is translated into German as Besser würden Sie diese Dateien backup by M2M100. Here, as can be seen, the source verb back up is incorrectly reported as-is in the target text, albeit without whitespace between the two components. A potential explanation for this phenomenon could be the fact that the English noun backup can be translated into German with the noun Backup (or Back-up), which the model could have learned during the training phase. However, in the source text provided, the word back up is a verb and should have been translated with sichern or eine Sicherungskopie machen.

Another noteworthy instance of this type concerns the verb crash which is borrowed from English into Dutch as crashen. For example, the word crash in the source text Youcan crash here, though it’s not very comfortable is reported as-is in the Dutch translation, while the correct infinitive form crashen should have been used. Generally, based on our manual analysis, M2M100 seems particularly prone to this behavior in Dutch, leaving words such as brass, chap, brick, catch, check out, and clean up untranslated.

Finally, this error category also includes those cases in which the focus words are simply transliterated into a different alphabet. We observe this phenomenon in Bulgarian, for example, in the source sentence Okay, everyone sit on your bum and try and touch your toes, in which the focus word bum is merely transliterated into Cyrillic with бум.²⁴

In the field of MT, hallucinations can be described as incorrect translations that exhibit a severe semantic detachment from the source text (Lee et al. 2018). For the purposes of our analysis, we concentrate on hallucinations generated when translating the ambiguous focus word. Importantly, we differentiate between hallucinations and disambiguation errors. In fact, as illustrated previously, in disambiguation errors the focus word is translated with one of its senses, but not the correct one in the given context. Instead, hallucinations are not senses of the focus word.

Interestingly, upon manual inspection, hallucinations seem to occur more frequently when the focus word is a verb. As can be seen in Table 11, the source sentence The soldier acquitted herself well in battle is translated by M2M100 into Italian with Il soldato si è goduto bene nella battaglia, in which the Italian verb godersi is semantically detached from the English source word acquit. In fact, godersi means to enjoy, take pleasure in an activity, and does not lexicalize any meaning expressed by the verb to acquit, which, instead, according to WordNet, can refer to: (i) pronounce not guilty of criminal charges; and (ii) behave in a certain manner. Interestingly, we observe another hallucination produced by the same model which translates the same source text into Spanish with El soldado se acercó bien en la batalla. Here, the Spanish verb acercarse (i.e., to go/move closer, among other meanings) cannot convey the meaning of the focus word in the source text. Additional noteworthy hallucinations are observed in Bulgarian, where the source text render fat in a casserole is translated into Bulgarian by M2M100 with Нанесетемазнина в касис which could be translated literally as Apply fat to currants. Finally, the focus word in the source text The dog’s laps were warm and wet is translated into Italian with the equivalent of paws by GPT-4.

We identify a specific type of hallucination including words that do not exist in the vocabulary of a given target language. Among the causes for non-existing target words, we observe orthographic errors, incorrect morphological modifications of an existing word, and the creation of a word or expression that cannot be understood by a native speaker. Interestingly, this type of error may also originate from (pseudo-)loanwords, that is, words (apparently) adopted from a foreign language other than the source or the target one with little or no modification. Examples of non-existing target words can be found in Table 12.

In Dutch, we observe a few non-existing target words connected to orthographic errors. For example, the Dutch translation of the English verb back up is misspelled by various models such as MBart50 in You’d better back up these files translated as Je zou deze files beter back-upen, while the correct spelling is back-uppen. Similarly, in Slovene we identify several spelling errors. For instance, the source sentence Back up the car a little, you’re blocking the driveway is translated by MBart50_mtm with Če malo vzpenemo avto, zapremo prikolinec in which the word form vzpenemo does not exist in Slovene and the correct spelling would be vzpnemo. As a matter of interest, we note some instances in which MBart50_mtm creates new words in Slovene by prepending an existing prefix to an existing verb in Slovene, which, however, results in a non-existing word, such as razbrisati, composed of the prefix raz, generally conveying the meaning of separation, dispersal, or spreading apart, and the existing verb brisati, which means to wipe, erase, or delete, in the translation razbrisati se na sestanku…Stol je bolan.

We observe a significant number of cases of linguistic interference in which the translation of the focus word is or seems to be derived from a language other than the source and the target one. We refer to these cases as (pseudo-)loanwords. For instance, the source text Though initially adopting a hard-line stance, the politician soon started to backpedal is incorrectly translated into Spanish by M2M100 as Aunque inicialmente adoptó una postura de línea dura, el político pronto comenzó a retrocedir in which the word retrocedir is a Catalan verb, while the corresponding word in Spanish is retroceder.

In Dutch, we identify several instances in which the source word is translated either with a German word or with a word that is very similar to a German word. For instance, the source text The press gang used to impress people into the Navy is translated by MBart50 as De persbank beeindruckde mensen in de marine in which the word beeindruckde does not exist in Dutch and seems to be derived from the German verb beeindrucken (even though the form proposed by MBart50 does not exist in German either). Instead, the sentence He placed his hands on the arm rests of the chair is translated by MBart50 with Hij zette zijn handen op de armstützen van de stoel where Armstütze is a German word meaning armrest.

Similar cases can be found in Slovene as well, where we observe some instances in which the source word is translated into a language other than Slovene. For instance, we identify this phenomenon in the following source sentences: (i) Freud thought of cathexis as a psychic analog of an electrical charge in which MBart50_mtm translates the focus word with павантаження, which resembles the Ukrainian word навантаження meaning load; (ii) If the goalkeeper is injured, we have a backup, in which MBart50 uses the Czech word záloha, which can be translated into English with advance or backup; (iii) Though initially adopting a hard-line stance, the politician soon started to backpedal, where the focus verb is translated by MBart50_mtm as odstupati, which is a Serbian and Croatian word.

Compared to commercial MT models and larger LLMs, we observe that the smaller LLMs exhibit an overall significant decrease in translation quality and an increase in MISS instances. Given the higher percentage of MISS cases reported by smaller general-purpose open-source LLMs, we now investigate the error patterns identified when analyzing the output of such models. In general, we observe that general-purpose LLMs show many cases of omissions, untranslated source words, and hallucinations.

As far as omissions are concerned, we note several cases in which models fail to provide any translation and, instead, offer an explanation for why the translation cannot be generated. For instance, when prompted to translate the source text They tracked him back toward the head of the stream into Italian, Gemma-2B answers with I cannot translate the text to Italian without context, as the text does not provide any context. Along these lines, when required to translate Lines of communication were set up between the two firms into Italian, Gemma-2B answers with I cannot translate the text as it does not provide any context or information about the two firms or their communication. Similar cases can be observed when translating into German (e.g., when receiving as input the source text He got life for killing the guard), the same model outputs I cannot translate this sentence as it contains a violent and harmful statement. I am not able to promote or endorse violence or incite hatred, or when asked to translate the sentence Pull a bank robbery into Bulgarian, Gemma-2B answers: I cannot translate that sentence, as it is not appropriate to talk about illegal activities. Other cases of omission impact only the focus word. For instance, the source text His trousers bag at the knees is translated into Spanish by Phi3-mini as Sus pantalones en las rodillas, where the focus word bag is omitted.

Interestingly, we observe untranslated source text when analyzing the output produced by various models, for example, when instructing LLaMA2-7B to translate the source text His face has many lines. In some other cases, we observe that LLaMA2-7B leaves part of the source text, including the focus word, untranslated, for instance, the source text Let me show you my etchings is a rather worn line is translated with Могуподелиться с тобой своими этчéжами - этоrather worn line.

Finally, among the several cases of hallucinations, we note a substantial number of non-existing target words. For instance, Phi3-mini translates the source text Alternationof summer and winter into Italian with Alternatazionedi estate e inverno, where the word Alternatazione does not exist in Italian. Similarly, the same model translates the source text Hebore himselfwith dignity into Spanish with Portróbalecon dignidad, containing the word Portróbale which does not exist in Spanish. The source text Hey, buddy, you got a light? is rendered into Russian as Привет, товарищ, у тебя естьликет?, where the word ликет does not exist in standard Russian. Another noteworthy hallucination is produced by LLaMA2-7B which translates the source text The office was full of secretheads with Ofice was full of tajnye glavy, where the word glavy does not exist as-is in Russian and corresponds to the transliteration of the word главы. Finally, several hallucinations are observed when translating into Bulgarian, for example, I’m sitting for a painter this evening is rendered by Gemma-2B as се за паметник тази вечер, where the word Сяням is not a translation equivalent for sitting.

While the previous section provided crucial insights into the output of MT systems by manually analyzing the errors produced when dealing with lexical ambiguity, we now move on to a computational inspection and assessment of the disambiguation capabilities of such systems. To this end, we propose an extensive array of experiments aimed at exploring five of the seven research questions we introduced in Section 1. First, we concentrate on the role played by the encoder in dealing with ambiguous words. To this end, we first study the correlation between the disambiguation capabilities of the encoder and those of the entire architecture (RQ3). We then evaluate the effectiveness of the latent representations produced by the encoder for disambiguation purposes. Next, we move on to assess the relationship between the capacity of an architecture and its disambiguation capability (RQ4). We posit that multilingual models tend to allocate more resources to learning proper representations for common senses across languages, thus compromising their ability to deal with infrequent senses, a phenomenon which we refer to as budgeting. We explore this hypothesis by investigating whether multilingual MT systems sacrifice their disambiguation performance in order to be able to handle multiple languages (RQ5). Furthermore, we study the impact of the decoding strategy on the MT systems’ disambiguation capabilities (RQ6). Finally, we explore the extent to which standard MT evaluation settings are suitable for detecting disambiguation errors by comparing them to the test scenario proposed in our work (RQ7). Given the need to access models and their components directly, the aforementioned experiments are conducted using the open-source, encoder-decoder MT models illustrated in Section 4.1.

Over the last few years, the most common architecture used for MT systems has been the Transformer (Vaswani et al. 2017), in which—except for GPT-like approaches—an encoder takes as input the source sentence, and a decoder generates its translation autoregressively. The decoder is designed to attend only to the latent representations produced by the last layer of the encoder module and the representations from its prior decoding steps, without directly accessing the input source sentence. Hence, intuitively, such representations should encode semantic information regarding the source words and their meanings. In order to study the contribution of the encoder to distinguishing word senses (RQ3), we freeze the encoders of several MT systems and use them to extract the representation of the source sentence and provide it as input to a two-layer fully connected neural classifier trained to perform WSD. Subsequently, we compare the performance obtained by this classifier with the DiBiMT accuracy (see Section 3.4) of the entire architecture, and measure their correlation.

We train a two-layer, fully connected neural classifier to perform WSD taking MT systems’ encoders’ representations as input. We train these systems on the concatenation of standard WSD training datasets, that is, SemCor (Miller et al. 1993) and the Princeton WordNet Gloss Corpus (Langone, Haskell, and Miller 2004, WNG).²⁵,²⁶ We use SemEval-2007 (Pradhan et al. 2007) as our development set.²⁷ As far as the test data is concerned, we measure the disambiguation performance of our systems on the sense-tagged sentences contained in the DiBiMT dataset, which can also be used as a benchmark for WSD, since each instance is manually annotated with the most suitable BabelNet sense of the ambiguous focus word. However, a direct comparison between the disambiguation performance and that obtained against DiBiMT would not be fair if computed on all sentences, since the accuracy in DiBiMT considers only the sentences classified as either GOOD or BAD, as described in Section 3.4. Therefore, in order to level the playing field, for each model, we only compute the WSD accuracy on the set of sentences classified as either GOOD or BAD according to DiBiMT.

We show the result of this experiment in Figure 5. We measure a statistically significant Pearson correlation of 0.95 between WSD and DiBiMT accuracy scores, with a p-value <0.001, suggesting that, as expected, the decoder is using the encoder’s last layer representations in order to disambiguate. Furthermore, we note that in some cases—specifically, for M2M100 and M2M100_lg—the WSD accuracy of the encoder is higher than the corresponding DiBiMT accuracy, which could indicate that the decoder is not always capable of successfully leveraging all the information embedded in the representations provided by the encoder. Interestingly, although most models demonstrate comparable accuracy in both WSD and DiBiMT, this trend is not observed for the models of the OPUS family, where the DiBiMT accuracy is substantially higher. While this may be due to the smaller number of parameters of their fully connected classifier,²⁸ we argue that there might be other reasons for this behavior. In the following sections, we delve deeper into the analysis of the representations produced by the encoder and also provide additional reasons to help understand why OPUS models show a higher DiBiMT accuracy compared to that achieved in the WSD setting.

In the previous section, we showed that the disambiguation capabilities of the encoders correlate strongly with those achieved by the corresponding architectures, when evaluated against the DiBiMT benchmark. We now study the disambiguation performance obtained by relying on the representations produced by the encoders, also in relation to the capacity of the corresponding systems (RQ4). Specifically, we compare the systems’ performance to that achieved by an encoder-only pre-trained language model (PLM), that is, BERT (Devlin et al. 2019), since we expect the encoder of MT models to produce vector representations that have different properties compared to those of PLMs. In fact, while the MT training objective requires models to implicitly capture the meaning of words in order to translate them correctly, BERT was trained on the Masked Language Modeling objective, which explicitly requires models to learn rich contextual representations of a word based on the context in which it appears.

The experimental setup is partially shared with that of the previous experiment. Specifically, we train the neural classifiers on the datasets illustrated in Section 6.1.1. Instead, as for the evaluation, we test systems on two different datasets:

• ALL_new (Maru et al. 2022), a refined version of the ALL Senseval and SemEval standard WSD dataset collection (Raganato, Camacho-Collados, and Navigli 2017);

• DiBiMT_wsd, that is, the sense-annotated sentences contained in our framework. However, differently from the previous experiment we do not restrict the set of sentences used for this experiment, employing instead the full set of sense annotations as we do not need to compare our results with the DiBiMT scores of the entire architectures.

We compare all open-source MT models with the following baselines and systems:

1. Most Frequent Sense (MFS): This baseline disambiguates an ambiguous word by predicting the sense occurring most frequently in SemCor, as customary in WSD;

2. Random choice: This baseline predicts a sense uniformly at random among the candidates of a word in context;

3. BERT_lg: We train the same two-layer fully connected neural classifier as in Section 6.1.1 on the output of a BERT-large model;

4. BERT_lg random: We train the neural classifier as above, but by randomly initializing and freezing the parameters of the BERT-large model. We use this baseline as a lower bound, as the representations extracted from the encoder hold no particular meaning, making it difficult even to discriminate between different words, let alone different senses of the same word. Therefore, we expect that the performance obtained by this baseline can be completely attributed to the ability of the classification head to learn simple patterns (e.g., learning to predict the MFS).

We report the results of this experiment in Table 13. The difference in performance between models is not particularly evident on ALL_new, where the systems are in the same ballpark, with the sole exception of the smaller OPUS models. Instead, significant performance gaps can be observed when considering the results obtained on DiBiMT_wsd, where only the bigger NLLB-200 models are able to achieve a performance above or on par with BERT_lg, and all other MT models perform worse than BERT_lg by a sizeable margin. In this respect, Figure 6 shows how the WSD accuracy on DiBiMT_wsd varies in relation to the capacity of the models. Several tested encoders are larger than BERT_lg, with NLLB-200_lg featuring more than 5 times its parameters; nevertheless, similar-size M2M100_lg and MBart50 models attain an accuracy score that is considerably lower than that of BERT_lg. These results suggest that the pre-training objective of BERT_lg is more suited for the task of WSD. Furthermore, it seems that, while more capacity is key to producing richer representations, the training recipe can make a big difference, given that the smaller NLLB-200_sm model outperforms several other MT encoders, achieving a higher score than the bigger M2M100_lg, when evaluating against DiBiMT_wsd.

Looking at the left-hand side of Figure 6, instead, the OPUS models exhibit remarkably low accuracy. In light of the performance of the baselines, it is particularly interesting to look at OPUS_mul, which obtains an accuracy score on DiBiMT_wsd that is only slightly above the lower bound provided by BERT_lg random. We hypothesize that the representations of its encoder are not rich enough to provide a decision boundary for infrequent senses, which is further demonstrated by the performance of the same model on ALL_new, where OPUS_mul scores well above BERT_lg random. This is probably due to the focus words in DiBiMT being more difficult to disambiguate since these show a higher polysemy degree,²⁹ and the most frequent sense is more rarely the correct one; this is also confirmed by the performance of the MFS and Random choice baselines. Indeed, (i) MFS attains a relatively decent score on ALL_new, but a particularly low one on DiBiMT_wsd, indicating that the most frequent sense is seldom the correct one for DiBiMT_wsd instances; and (ii) the performance of the Random choice baseline decreases moving from ALL_new to DiBiMT_wsd, which is due to the ambiguous focus words in DiBiMT_wsd having a higher polysemy degree, and therefore more candidate senses to choose from.

Finally, we offer a partial answer to the question raised in Section 6.1.2: Why do OPUS models demonstrate superior DiBiMT performance compared to their encoder’s WSD accuracy? Since OPUS_mul performs similarly to BERT_lg random, we hypothesize that the decoder compensates for the encoder’s limited contextualization by leveraging its own parameters to generate correct lexicalizations. Instead, OPUS_bil shows a distinct pattern, as its encoder’s WSD accuracy on DiBiMT is significantly higher than BERT_lg random, suggesting that there might be a different reason behind its performance. In the next section, we investigate the phenomenon of budgeting under the lens of the performance difference between OPUS_bil and OPUS_mul, and put forward an explanation for this behavior.

In the context of studying disambiguation biases in MT, we use the term budgeting to refer to the phenomenon whereby a model sacrifices its ability to represent infrequent senses of ambiguous words so as to be able to learn other information, such as that coming from more languages. On the basis of the surprisingly high performance of OPUS_bil compared to its size and that of the other models tested, we hypothesize that multilingual models may be budgeting their disambiguation capabilities to memorize more common senses in multiple languages. In this section, we investigate this phenomenon by analyzing the performance obtained by OPUS_bil and OPUS_mul, thereby answering RQ5. We note that this experiment shares the experimental setup with that of Section 6.2.

Table 14 reports the disambiguation performance achieved by using the encoder representations of OPUS_bil and OPUS_mul, both on ALL_new and DiBiMT_wsd, as well as the DiBiMT accuracy obtained by the corresponding architecture (column DiBiMT_mt). In every tested language, the performance of OPUS_mul is worse than that of OPUS_bil. As mentioned in Section 6.2, the performance gap is emphasized when evaluating against DiBiMT sentences, where OPUS_bil models obtain a score that is significantly higher than that achieved by OPUS_mul. We highlight that OPUS_mul features the same neural architecture and a number of parameters comparable to that of OPUS_bil, and, to the best of our knowledge, OPUS_mul was trained using the data of the bilingual models combined. Therefore, we attribute its worse performance to the phenomenon of budgeting which might hamper its disambiguation capabilities.

Going back to the question posed in Section 4.4, that is, Why does OPUS_biloutperform larger models?, budgeting provides a possible answer. In fact, all M2M100 and MBart50 models are massively multilingual, thus using the same set of parameters to translate multiple languages. However, while the results of this experiment suggest that budgeting plays a significant role in this situation, we cannot isolate this phenomenon as the only reason for the performance disparity, since M2M100 and MBart50 were trained using different datasets and algorithms compared to OPUS_bil.

Furthermore, we hypothesize that budgeting is also responsible for lowering MT models’ DiBiMT accuracy to the level of their corresponding encoder’s WSD accuracy, as discussed in Section 6.1.2. Indeed, the decoder of a multilingual model is required to learn to generate text in several output languages, which might reduce the model’s capacity allocated to accurately translating rarer senses. Conversely, since OPUS_bil’s decoder is asked to generate text in a single language, additional capacity can be allocated to compensate for the limitations of its encoder’s representations, thus attaining a DiBiMT score higher than its encoder’s WSD performance.

Given the copious amounts of data on which models are trained, it might be possible that the information necessary to translate infrequent senses is stored latently somewhere within the model. Since MT systems generally rely on the beam search algorithm with relatively low beam sizes to produce translations, an insufficient exploration of the decoding tree could prevent models from choosing a sequence containing a correct translation. In order to investigate this possibility, we now tackle RQ6, that is, the impact of the beam search on the disambiguation capabilities of MT models. Specifically, we investigate whether the decoding strategy utilized by an MT model can be considered, at least partially, responsible for its disambiguation errors. To this end, we explore the decoding space of MT models using the beam search algorithm with several beam size values β, that is, 1 (equivalent to greedy decoding), 5, 20, and 100, and with default generation parameters, in the following two different settings.³⁰

Table 15 reports the DiBiMT accuracy as computed on the min-perplexity translations produced by beam search at each value of β. Unexpectedly, we find that increasing the beam size does not guarantee the min-perplexity sentence to have a lower perplexity than those found in smaller beam sizes. Therefore, we also compute accuracy using the translations with the absolute lowest perplexity which we could find in any of the 126 explored beams (column Min).

Table 16 reports the maximum DiBiMT accuracy that can be obtained by cherry-picking the best translations at each value of β (i.e., as if we had an oracle telling us whether or not a translation is correct). That is, given a list of translations decoded by a model for a specific instance in DiBiMT, that instance is considered GOOD if any of the decoded translations is classified as GOOD, while it is considered BAD if there are no GOOD translations and at least one BAD translation, and MISS otherwise. We also compute the oracle accuracy using, for each item, the best of the 126 explored beams (column Comb).

We observe that, using the oracle, the performance of every model steadily increases with larger beam sizes: On the one hand, we observe that, at least in this setting, models are not able to decode any GOOD translation in a high percentage of cases (i.e., from approximately 32% for NLLB-200_mdd to almost 75% for OPUS_mul); on the other hand, this indicates that by exploring the decoding space of the models we are indeed able to find GOOD translations that had not been found with smaller beam sizes, with an average accuracy increase of approximately 15 points between β = 1 and β = 100. Nonetheless, by looking at the results in Table 15, we notice that the DiBiMT scores do not significantly improve when selecting min-perplexity sentences found by exploring larger beams. This suggests that, regardless of the beam size, these MT models tend to be more confident in producing sentences containing a BAD translation of the focus word rather than a GOOD one. These results indicate that the evaluated MT models display an intrinsic bias toward more frequent senses of ambiguous words. Interestingly, we also notice that increasing the beam size does not always steer the generation toward sentences that are deemed more probable by the underlying model. Indeed, we found that, in approximately 64% of cases, the most likely translation is not generated using a beam size of 100. In particular, by looking at column Min of Table 15, we notice that some systems have slightly higher accuracy scores when we select the most probable translation among all generations. This suggests that whenever aiming to obtain the generation deemed most probable by the model via beam search, increasing the beam size as much as possible might not be the best option. Instead, it might be more effective to generate several sets of sentences with different beam sizes, and then return the most probable one among all generations.

As a matter of interest, when considering any of the translations produced by any beam size in the oracle setting (column Comb in Table 16), the results remain almost identical to β = 100, indicating that, most of the time, at least one among the 100 decoded sentences contains a GOOD translation of the focus words.

Arguably the best way of evaluating MT systems is to ask professional human translators to rate their outputs. Indeed, over the last few years, several techniques for human evaluation have been proposed, where annotators are tasked with either rating the quality of translations with a score between 1 and 100 (Graham et al. 2013; Kocmi et al. 2022), or with identifying and classifying the category and severity of error spans in the translations (Lommel, Uszkoreit, and Burchardt 2014; Freitag et al. 2021). However, human evaluation is expensive, and therefore difficult to use for assessing the quality of different iterations of the same models, let alone for selecting the best checkpoints at training time. In order to obtain a less expensive and time-consuming evaluation, the community relies on automatic evaluation strategies. These are based on one or more evaluation metrics that assess the degree of faithfulness and adherence of a candidate translation to the corresponding source text.

We now aim to determine if a standard automatic MT evaluation setting is effective at detecting disambiguation errors (RQ7), and whether it can be improved by accompanying the assessments with the scores returned by a dedicated benchmark like DiBiMT. To this end, we select two widely used test datasets, along with four among the most popular evaluation metrics.

We first translate the sentences of the test datasets using several MT systems, then we score their translations using the selected metrics and measure their correlation with the DiBiMT accuracy. Our goal is to understand whether the MT systems that are ranked higher by the DiBiMT accuracy—and therefore more capable of disambiguating ambiguous words—are also ranked higher by popular MT metrics on the selected test sets. Therefore, we use Kendall’s tau coefficient, which is a statistic used to measure the ordinal association between two measured quantities (Kendall 1938). In addition, we qualitatively analyze our results to get a sense of how current evaluation techniques fare when dealing with systems that have different disambiguation capabilities.

We measure the performance of our MT systems on two test datasets:

1. Flores-200 (Goyal et al. 2022; NLLB Team et al. 2022): A parallel corpus containing 3,001 sentences coming from English Wikipedia³¹ and translated into 200 languages by professional translators. The dataset is divided into three splits: dev, devtest, and test, which is hidden. We report our scores on the devtest split.

2. Medline-2022 (Neves et al. 2022): A parallel corpus composed of abstracts of scientific publications in the biomedical domain retrieved from the MEDLINE database,³² along with five clinical case reports selected from publications of the Journal of Medical Case Reports, covering several translation directions: $en↔{es,it,de,ru,zh,fr,pt}$⁠.

We select Flores-200 and Medline-2022 because both have become standard benchmarks for measuring the performance of multilingual MT systems, the former for its wide coverage in terms of language pairs, and the latter for its domain specificity. Unfortunately, Medline-2022 covers only five out of the eight language pairs available in the DiBiMT benchmark.

We measure the performance of our models with four commonly used MT evaluation metrics:

1. BLEU is a precision-oriented metric computed using the number of overlapping n-grams between a translation and its reference (Papineni et al. 2002). The final score also takes into account a brevity penalty. We compute BLEU using corpus_score from the sacreBLEU library (Post 2018).

2. chrF++ compares a translation and its reference based on the number of overlapping character n-grams and word unigrams and bigrams they share (Popović 2015, 2017). We also use sacreBLEU for computing chrf++, using the corpus_score function.

3. BERTScore leverages pre-trained encoders to extract the contextualized embeddings of the tokens of a translation and its reference (Zhang et al. 2020). Then, it computes the cosine similarity between each pair of embeddings, greedily matching the most similar ones. Based on the embeddings’ similarities, BERTscore returns a Precision, Recall, and F1 measure. We report the F1 score, computed using the evaluate library from HuggingFace.³³

4. COMET is a machine-learned metric which takes as input a candidate translation, its reference, and the source sentence in the original language. COMET is trained with a regression objective to approximate human judgment (Rei et al. 2022), and outputs scores between 0 and 1, with 1 indicating a perfect translation. We use the default model Unbabel/wmt22-comet-da, which is based upon XLM-R (Conneau et al. 2020).

Table 17 presents the Kendall correlation between various metrics assessments and the DiBiMT accuracy for each target language. With the exception of the translation direction $en→zh$⁠, most correlations are notably high, indicating that current MT evaluation metrics tend to rank MT models with stronger disambiguation capabilities higher. However, this is not sufficient for answering RQ7. More specifically, it is probable that MT models that are proficient in translating highly ambiguous words inherently perform better overall, leading to higher scores from MT metrics. To explore this further, Figure 7 compares the metrics scores and the DiBiMT accuracy, both averaged across languages. Despite the high correlations, we observe significant differences in the value ranges between the metrics scores and DiBiMT accuracy. Specifically, the metrics scores tend to be relatively flat, suggesting that, while they generally rank systems similarly to DiBiMT, they do not clearly reflect the performance differences between them in terms of disambiguation capabilities. This is particularly evident when examining NLLB-200 models, where the DiBiMT accuracy increases from NLLB-200_sm to NLLB-200_lg by a substantial delta of approximately 15 points, while all metrics report modest improvements. Furthermore, most metrics rank M2M100 differently compared to DiBiMT, suggesting that M2M100 may particularly struggle with translating ambiguous words while at the same time being capable of generating overall good quality translations.

As a final consideration, we highlight that MT metrics consider various factors for assessing overall translation quality, which might lead to overlooking or not sufficiently penalizing disambiguation errors. Additionally, when compared with the sentences in the DiBiMT benchmark, the lower ambiguity in the two considered test sets could be a factor. In conclusion, whether it be due to limitations in the metrics or the datasets, our results suggest that the current common evaluation setting does not probe MT models’ ability to translate ambiguous words effectively, highlighting the necessity for a benchmark specifically designed for it, such as DiBiMT.

Here we summarize the main findings of this section:

• The encoder of a Transformer-based encoder-decoder MT system provides a critical contribution to the overall disambiguation capabilities.

• In terms of disambiguation performance, MT systems’ encoders learn representations that are less effective compared to BERT. However, this gap diminishes or even disappears with the use of larger MT models.

• Multilingual models seem to trade off their disambiguation capabilities in order to memorize more common senses in multiple languages.

• MT systems are intrinsically biased toward more frequent senses of ambiguous words, and increasing the beam size does not guarantee better disambiguation performance.

• MT evaluation needs an ad-hoc benchmark to assess how systems fare with the translation of ambiguous words, as the standard setting of evaluation is not fitting to be employed for investigating this phenomenon.

In this section, we put forward some recommendations to the community for enhancing the MT capabilities based on the findings of our work:

• Going beyond the surface and exploring the understanding capabilities of MT: While MT systems have achieved tremendous progress in producing high-quality translations, this work highlights the crucial need to focus on their ability to deal with non-predominant meanings of ambiguous words. While some works already tackle this issue (Iyer, Chen, and Birch 2023; Iyer et al. 2023), there is still a long way to go to ensure that lexical ambiguity is properly addressed in MT.

• Higher-resource languages still deserve attention: While substantial efforts are being made to improve the availability of parallel corpora for low-resource languages, we observe that, despite its low number of MISS instances, a high-resource language such as Dutch exhibits the poorest performance in DiBiMT (see Table 5). In light of this counterintuitive finding, we encourage the MT community to assess and address the impact of lexical ambiguity on high-resource languages as well, because these, unexpectedly, may pose significant challenges.

• Studying the interplay between LLMs and MT systems: We observe a recent trend in the literature suggesting that adapting LLMs may yield superior results compared with training traditional MT architectures from scratch, while dropping the requirement of vast amounts of parallel corpora for training (Alves et al. 2024; Xu et al. 2024). In our work, we further explore this trend by assessing LLMs’ ability to deal with lexical ambiguity. Interestingly, our experimental results show that Tower-7B, a language model for translation-related tasks built upon LLaMA 2, which we experimented with, outperforms LLaMA2-7B, LLaMA3-8B, and all open-source MT systems when evaluated against DiBiMT. In light of such findings, we suggest that the research community further investigates the potential of LLMs in MT.

• Re-balancing the training data of MT systems: One of the crucial obstacles that systems face when dealing with lexical ambiguity is sense infrequency. To translate words used with infrequent senses effectively, we encourage the community to explore artificially increasing the occurrences of such senses in training corpora. However, such data modification could compromise the systems’ ability to properly translate common senses. In such experimental scenarios, the DiBiMT benchmark can play a pivotal role in evaluating the effects of any adjustments to the training methodology and achieving an optimal balance.

In this article, we present DiBiMT, an entirely manually curated benchmark for investigating the ability of MT systems to deal with lexical ambiguity. DiBiMT covers eight language pairs, each composed of English and one of the following languages: Bulgarian, Chinese, Dutch, German, Italian, Russian, Slovene, and Spanish. We put forward a detailed study of the impact of lexical ambiguity on the automatic translation output by 22 systems, including both commercial and non-commercial MT systems, two commercial LLMs, and a set of open-access LLMs. We find that commercial systems consistently outperform their open-access counterparts, with GPT-4 achieving the best results overall. Among the open-access systems, Gemma2-9B stands out, achieving results that are substantially higher than all the rest and close to GPT-4. Moreover, although the performance of all systems significantly decreases when dealing with infrequent meanings and highly ambiguous words, we notice that top-performing models are more robust in handling the latter phenomenon. Conversely, all models struggle similarly with sense infrequency, which proves to be the most challenging obstacle in disambiguating ambiguous words. In this respect, we find that systems translate the focus word with the most frequent sense in about 50% of cases, while approximately 75% of the time they use a sense that is more frequent than the correct one. In light of these findings, and to counter the most prominent obstacle of sense infrequency, we put forward promising research directions for enhancing MT capabilities. Among our recommendations for future improvements, we include adapting LLMs to MT, since such systems do not necessarily require vast amounts of parallel corpora as traditional MT systems do. Furthermore, we suggest investigating and addressing the impact of lexical ambiguity on higher-resource languages, since these might pose unexpected challenges. We study the nature of errors produced by systems when translating ambiguous words by carrying out a manual error analysis, in which we identify four error patterns, namely, disambiguation errors, omissions, untranslated source words, and hallucinations. The first error category, disambiguation errors, is the most relevant to the DiBiMT benchmark, since this pattern consists of incorrect translations due to the erroneous choice of word meanings. Instead, omissions occur when a given model does not translate the focus word. We differentiate between severe and mild omissions depending on the impact of the omission on the overall quality and comprehensibility of the output translation. The third error pattern is that of untranslated source words, which are focus words reported as-is and therefore not translated into the target language. Finally, the last error pattern is represented by hallucinations, consisting of translations showing a significant semantic detachment from the source text. Differently from disambiguation errors, hallucinations are not possible senses of the focus word. Hallucinations also include a sub-pattern which we refer to as non-existing target words, that is, output words and expressions that do not exist in the vocabulary of the target language. Interestingly, based on our manual inspection, we find that, while disambiguation errors and hallucinations are commonly found in the output translations of all systems considered, untranslated source words and non-existing target words almost exclusively affect non-commercial systems.

In order to better investigate the ability of systems to deal with lexical ambiguity, we also carry out an extensive array of experiments. First, we investigate the impact of the encoder on the ability of a given MT model to disambiguate ambiguous words. Our results demonstrate that the encoder provides a vital contribution to the disambiguation capabilities of the corresponding architecture. Furthermore, we find that the representations learned by BERT are more effective than those learned by the encoders of MT systems when dealing with ambiguous words. Nonetheless, we show that capacity is key, and the encoders of larger MT systems are capable of surpassing BERT. We also investigate the presence of a trade-off between multilinguality and performance, a phenomenon which we refer to as budgeting, by evaluating a multilingual model and its bilingual counterparts against the DiBiMT benchmark. We find that the multilingual model OPUS_mul consistently achieves worse disambiguation performance than its bilingual counterparts. We then examine the role of the decoding strategy in translating ambiguous words, focusing on the impact of varying beam sizes on a model’s ability to disambiguate these words. Our experiments reveal that an increase in beam size does not necessarily improve performance, potentially because models are, in a high percentage of cases, not able to translate the focus word altogether. In this context, we show that MT systems exhibit an intrinsic bias toward more frequent senses of ambiguous words. Specifically, these systems might assign a higher probability to wrong translations of the ambiguous focus word despite the pool of decoded translations also including correct translations. Finally, we investigate the effectiveness of current MT evaluation metrics such as BLEU, chrF++, BERTScore, and COMET in assessing the ability of models to disambiguate ambiguous words, measuring the performance of MT systems on two popular test sets, namely, Flores-200 and Medline-2022. Crucially, we show that the standard evaluation setting is not fitting to be used for the investigation of disambiguation errors—whether it be due to the metrics not capturing them, or to the test sets not being particularly suited for the task—and highlight the need for an ad-hoc manually curated benchmark such as DiBiMT.

Our benchmark is available at https://nlp.uniroma1.it/dibimt/. In light of the major impact of lexical ambiguity on machine-generated translations, and despite the efforts made so far, we encourage the research community to devote particular care to improving the disambiguation capabilities of MT systems, thereby enabling a significantly better translation performance.

This work aims to create a novel entirely manually-curated evaluation benchmark called DiBiMT which allows semantic biases in MT to be investigated.

With this aim in view, you receive a spreadsheet that contains approximately 1,000 automatically-extracted instances, each comprising the following data: i) a lemma and its part of speech (PoS), associated with a definition derived from either WordNet or Wiktionary; ii) a sentence in English containing a focus word for which some good and bad translation candidates derived from BabelNet are provided.

For each instance, good translation candidates are located on the same line as the definition and the example, whereas bad translation candidates can be found on the line below. From a translation perspective, a good candidate can be described as a correct translation into the target language for the English focus word. Instead, a bad candidate is an incorrect translation for the English focus word in the given context.

Annotators are asked to verify: i) the correctness of the good translation candidates and add new good translations if deemed necessary; ii) the incorrectness of the bad translation candidates provided. Furthermore, annotators are required to adopt the following guidelines. Do not annotate idioms and mark them with the tag IDIOM. Do not annotate instances in which the semantic context does not allow us to unequivocally determine the meaning of the focus word and label these with the tag X. Discard instances containing proper names as focus words (e.g., The military campaign near that creek was known as “The battle of Bull *Run*”). Mark with the tag DISCUSS challenging instances which should be discussed during joint sessions. Annotators are allowed to include cross-PoS candidates, that is, candidates whose PoS is different from that of the focus word and, when this is the case, annotators are required to include the candidate in square brackets in the following way: [candidate_with_different_pos | P]. Annotators are asked to do the same for multi-word expressions as well: [multi_word_expression | P]. Annotators can specify the PoS by adding a letter (either, n, a, v or r) after | P.

Once the aforementioned annotation steps are finalized, annotators are required to study and classify as either GOOD or BAD all so-called MISS cases, that is, translation candidates proposed by the MT models and LLMs considered and classified as neither GOOD nor BAD. Please note that we expect that in only a small percentage of MISS cases is the target translation actually missing, such as in omissions (see Section 5).

As illustrated in Section 3.4, SFI weighs instances based on their ambiguous word meaning’s frequency index, that is, $μλP(σ)$⁠. This metric could be reformulated to use sense frequencies instead of their indices to weigh the item’s contribution to the metric score. In this section, we discuss our reasoning for using an index-based weighting instead of a frequency-based one.

Our assumption is that computing frequencies for very low-frequency meanings is inherently bound to produce a very noisy and unreliable estimation, especially when considering that there are very few manually annotated corpora for WSD.³⁴ Therefore, given that word meanings tend to follow a Zipfian distribution, and that our very infrequent senses are likely to have wrong estimations, we decide to leverage Zipf’s law and use the indices instead of the raw frequency, as it should display better stability across different corpora compared to raw frequency counts, providing a much more robust and reliable score.

In this section, we report the generation parameters used when translating the sentences of DiBiMT with the open-source MT systems; the API call we use for translating with DeepL, Google, GPT-3.5_turbo, and GPT-4; and the prompt provided to GPT systems and LLMs.

For open-source models, we use a beam size of 5, with early stopping. We apply no length penalty and return the sequence with the lowest perplexity, among those generated using the beam search algorithm.

We generate translations into all our target languages with GPT-3.5_turbo and GPT-4 using the API made available by OpenAI.³⁵ For both models, we use the following instruction-based prompt: “Translate the following English text to {lang}: {source:text}”, where {lang} and {source:text} are two variables containing a given target language and the source sentence to be translated, respectively. In both cases, we adopt the default hyperparameters.

In this section, we report additional information regarding the overall results of systems on the DiBiMT benchmark, discussed in Section 4.

A breakdown of UNK instances is reported in Table 18.

Table 19 reports the scores of MFS and MFS+ metrics, indicating the percentage of times that a system translates ambiguous words with their most frequent sense (MFS), or with senses more frequent than the correct one (MFS+). As can be seen from the Mean column, most systems obtain roughly the same scores, irrespective of their performance; indeed, even GPT-4, which is the best among the tested systems, reports MFS and MFS+ scores in line with the others. This suggests that almost all systems³⁶ tend to default to more frequent senses at the same rate.³⁷