High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics

Neural sequence-to-sequence models constitute the state-of-the-art for machine translation. These models estimate the probability of a target sentence given a source sentence. At inference, it is commonplace to approximate the maximum-a- posteriori (MAP) hypothesis with beam search in order to output a sentence with (close to) the highest probability given the provided source.

This strategy assumes that the sentences with the highest estimated probabilities should also be the translations with the highest quality as measured by humans. This assumption can be questioned based on two observations: (i) Neural Machine Translations (NMTs) generated by beam search are ranked below human translations in professional evaluations (Freitag et al., 2021a) while (ii) the NMT model itself considers human translations much less likely than its beam outputs (Ott et al., 2018). These observations clearly show that estimated probability and translation quality do not always correlate. An example is given in Table 1, where beam search generates a translation using mostly frequent words which results in inaccuracies. The two correct human translations contain infrequent words and phrases with low estimated probabilities based on the model.

These observations do not in themselves suggest an alternative to likelihood in selecting better hypotheses. For that, we look at recent progress in automated evaluation. Recently introduced utility metrics, such as Bleurt (Sellam et al., 2020a) or Comet (Rei et al., 2020), estimate human judgments u(h,r) from a candidate translation h and a reference human translation r with a neural network. These learned metrics have shown higher correlation with human judgments compared with traditional metrics based on lexical overlap such as Bleu (Papineni et al., 2002) and Meteor (Banerjee and Lavie, 2005). Bleurt and Comet have also been shown by the WMT metric task (Freitag et al., 2021b) to perform better than YiSi (Lo, 2020), which measures overlap in a neural embedding space. Bleurt and Comet are able to evaluate hypotheses with different word choices, sentence structures, and lengths compared to the reference translations. Unlike overlap-based metrics like Bleu, these metrics do not necessarily prefer the most likely tokens to increase the chance of covering n-grams in the reference translations (Freitag et al., 2020). When comparing a model output h and an alternative human reference r′, Bleu and Bleurt behave differently. While Bleu often estimates the quality of the model output h to be much higher than the alternative human translation r′ (Bleu(h,r) > Bleu(r′,r)), Bleurt and Comet typically prefer the human translation over the MT output (Bleurt(h,r) < Bleurt(r′,r)). This behavior generally agrees with professional raters (Toral, 2020).

These observations suggest that selecting model hypotheses likely to have a high quality score with respect to learned neural utility metrics should bring the quality of MT output closer to that of human translations. For that, we rely on Minimum Bayes Risk (MBR) decoding, in particular the sampling-based approximation recently introduced by Eikema and Aziz (2020). Sampling-based MBR starts with a set of unbiased samples drawn from an NMT model and finds the candidate which has the highest average utility when each hypothesis in the set is used as a pseudo-reference.

This MBR strategy has several potential pitfalls. First, the expectation of utility under the model distribution is used as a proxy to the expectation under the true underlying (human translator) distribution. This means that a high divergence between these two distributions will affect MBR (Pitfall 1: model quality). Second, the utility metric might be unreliable in areas of the space where it has not been evaluated (e.g., with low quality, low probability pseudo-references). This might cause its expectation to be very different from single point evaluations with high quality human references (Pitfall 2: utility validity over the reference space). Third, even if MBR discovers hypotheses with high utility with respect to actual human references, there is no guarantee that these hypotheses will receive high human judgments because these hypotheses are not necessarily close to the conditions for which the utility metrics have been designed (Pitfall 3: utility validity over the hypothesis space).

This paper evaluates MBR decoding for multiple utility functions and measures whether their predictions indeed improve the actual utility with respect to human references. We show that an NMT model based on the transformer-big architecture and Bleu, Chrf, Yisi, and Bleurt successfully avoid Pitfalls 1 and 2. We also study the robustness of these conclusions with respect to the number of considered samples and model size. We then conduct a human evaluation of MBR hypotheses with high estimated utility according to different metrics to assess Pitfall 3. We show that MBR decoding using Bleu as a utility metric slightly improves over beam search decoding, even though the difference between these two translations are minor. In contrast, MBR using Bleurt as a utility metric generates translations further away from beam output. These translations are given significantly higher human quality ratings compared with beam search and the other MBR hypotheses.

Our contributions are:

• We are the first to use neural metrics— Yisi and Bleurt—as utility functions during MBR decoding.

• We run a human evaluation with professional translators to assess the quality of MBR decode using different utilities.

• We show that MBR using Bleurt outperforms beam search decoding according to human judgments from experts.

• We further demonstrate that MBR decoding with Bleurt results in less likely translations which are lexically different from both beam output and MBR output relying on overlap-based utilities.

• We release all model hypotheses, candidate lists and human ratings as part of this paper.¹

Minimum Bayes Risk (MBR) decoding stems from statistical decision theory from the principal of maximization of expected utility (Bickel and Doksum, 1977; Berger, 1985). MBR has been applied to parsing (Goodman, 1996; Sima’an, 2003) and speech recognition (Stolcke et al., 1997; Goel and Byrne, 2000). The same idea was later applied to bilingual word alignment (Kumar and Byrne, 2002) and machine translation (Kumar and Byrne, 2004). MBR was used to maximize overlap metrics such as Bleu (Papineni et al., 2002) with statistical MT systems (Kumar and Byrne, 2004; Smith and Eisner, 2006; Tromble et al., 2008).

After the advent of neural machine translation (Sutskever et al., 2014), most methods relied on beam search to approximate MAP decoding (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). The question of optimizing utility metrics of interest such as Bleu was also explored. Approaches based on structured risk minimization (Edunov et al., 2018) or reinforcement learning (Bahdanau et al., 2017; Leblond et al., 2021) considered modifying the training procedure.

MBR decoding has recently gained attention in MT as a decision rule with the potential to overcome some of the biases of MAP decoding in NMT (Eikema and Aziz, 2020; Müller and Sennrich, 2021; Eikema and Aziz, 2021). While most prior work on MBR decoding for MT is based on k-best lists obtained via beam search, Eikema and Aziz (2020) proposed to use an approximation of MBR decoding based on unbiased sampling to overcome the shortcomings of MAP decoding. They demonstrated that samples from the NMT model are faithful to the training data statistics, while beam search is not. We adopt their sampling-based MBR decoding approximation in all our experiments.

The application of MBR to neural MT has focused on maximizing classical overlap-based metrics like Bleu, Meteor, Chrf, or Beer (Stanojević and Sima’an, 2014). Our work builds upon recent advances in the automatic evaluation of MT (Mathur et al., 2020), which has shown the emergence of learned utility metrics based on neural networks. We consider using neural metrics for MBR, which has not been done before. These metrics are neural networks that consider a pair of sentences (a hypothesis, and a reference) or a triplet of sentences (a source, a hypothesis and a reference) and output a real-valued score estimating the quality of the hypothesis. They rely on pre-trained monolingual or multilingual neural language models. The first generation of neural utility metrics uses neural models to extract pre-trained sentence and word representations to compute distances indicative of semantic proximity, for example, BertScore and Yisi (Zhang et al., 2020; Lo, 2019). Later, a second generation of neural utilities proposed to fine-tune neural models on human judgments, either through regression or ranking tasks. These approaches, such as Bleurt and Comet (Sellam et al., 2020a; Rei et al., 2020), have shown better correlation with human judgments (Mathur et al., 2020).

MBR relies on two essential components: a machine translation model and a utility metric. The translation model P_model(y|x) estimates the probability of any target segment y given a source segment x. The utility metric u(h,r) estimates quality of a candidate translation h given a reference translation r.

Growing ℋ_model is, however, computationally costly, both to obtain hypotheses and to evaluate their cross-utility. In all our experiments, we adopt the sampling-based approximation to MBR decoding (Eikema and Aziz, 2020) to generate a finite set of samples from a neural machine translation model. Eikema and Aziz (2020) showed that unbiased sampling provides a good approximation for the underlying model distribution. The cost of sampling is linear in the size of the set. Cross-utility can involve evaluating a large neural network as well and the cost of utility computation is generally quadratic in the size of the set. It is important to add that we generate independent samples, which implies that sentences with higher model probabilities have a higher chance to be drawn several times. By doing so and not deduping the candidate lists, we do not need to incorporate (again) the model probabilities during MBR decoding.

The automatic evaluation of machine translation is an active area of research (Mathur et al., 2020; Freitag et al., 2021b). MBR decoding centrally relies on a reference-based utility metric: Its goal is to identify a hypothesis with a high estimated utility (expectation under model distribution) with the hope that a high estimated utility translates into a high actual utility (with respect to a human reference), which itself should translate to a high human quality judgment. We experiment with utilities from different families of metrics:

Bleu (Papineni et al., 2002) measures lexical overlap as the geometric mean of the precision of n-gram matches with n ≤ 4 on the corpus level and adds a brevity penalty to penalize low recall hypotheses. As MBR decoding requires segment-level scores, we use add-one smoothed sentence-level Bleu (sBleu) (Lin and Och, 2004). during MBR decoding as an approximation. We use SacreBLEU (Post, 2018) for reporting corpus-level Bleu scores.²

We use Chrf (Popović, 2015) as an additional lexical overlap metric. Chrf uses character n-grams instead of word n-grams to compare the MT output with the reference. For Chrf we use the SacreBLEU sentence:chrf function (with default arguments³ ).

We also evaluate MBR decoding with neural utilities which has not been done before. We rely on Yisi-1-BERT (Lo, 2020) to represent first generation neural metrics, namely, metrics focusing on embedding-based overlap and not fine-tuned on human judgments. This metric relies on Bert (Devlin et al., 2019) to compute in-context word embeddings and then perform bi-directional alignments of n-gram matches in the embedding space to compute an F-score. For our experiments, we rely on base-cased Bert for English language evaluation and the multilingual model mBert for other languages. We use our in-house reimplementation of YiSi.

We rely on Bleurt to represent second generation neural metrics, that is, metrics not focusing on overlap but fine-tuned on human judgments instead. Bleurt is a regression model and relies on a learned embedding of the concatenation of the hypothesis and the reference translation. One of the strengths of Bleurt is that it can evaluate translations of different sentence structure, wording, and length in an unbiased fashion, as it is not focusing on any kind of overlap. This was one of our main motivations to revisit MBR decoding with neural metrics. We conducted experiments on two versions of Bleurt.

• Bleurt v0.1

  Bleurt v0.1 is a cased version of Bleurt (Sellam et al., 2020b) based on RemBERT (Chung et al., 2020). The model was pre-trained on more than 110 languages, and jointly fine-tuned on 13 target languages using the z-normalized WMT human evaluation data from 2015–2018.

• Bleurt v0.2

  Bleurt v0.2 is a joint model for all language pairs and is based on RemBERT. In addition to the fine-tuning data used for Bleurt v0.1, it also uses the WMT human evaluation data from 2019 and synthetic examples which consist of identities, alternative references, and random sentence pairs. Motivation for the latter was improved performance on very bad translations, a scenario frequently observed when scoring a candidate list during MBR decoding. Furthermore, instead of training Bleurt on the unbounded z-normalized scores, we manually scale them to a 0–1 range and clip the outliers.

We run experiments on two language pairs: English→German (En→De) and the reverse direction German→English (De→En) with models trained on WMT training data (Barrault et al., 2019). We use news-commentary-v15, paracrawl-v5.1, europarl-v10, and commoncrawl as training corpora with ∼57 million training examples after filtering out noisy data with contrastive data selection, as proposed by Wang et al. (2018). We also remove sentences longer than 250 tokens and sentence pairs with a source/target ratio exceeding 1.5. We use newstest2019 as our dev set to pick checkpoints and newstest2021 (Barrault et al., 2021) as our test set. For newstest2021, we have two reference translations (Ref-C and Ref-D for En→De and Ref-A and Ref-B for De→En).

We use the transformer implementation in lingvo (Shen et al., 2019), using a model similar to the transformer-big setting (Vaswani et al., 2017). The model has 6 encoder and 6 decoder layers, model dimension size of 1,024, hidden dimension size of 8,192, and the number of multi-attention heads is 16. Our models use a vocabulary of 32k subword units (Kudo and Richardson, 2018). We train the models until convergences for around 300,000 updates with a batch size of 43,000. We follow the suggestion of Eikema and Aziz (2020) and train our models without label smoothing. This slightly drops accuracy by 0.5 Bleu points on both language pairs when compared with a model using label smoothing. We run beam search with beam size of 4 and length penalty as described in Equation 10 in Wu et al. (2016) using α = 0.5. We do not use coverage penalty as this does not improve the results. For MBR decoding, we generate 1,000 unbiased samples for each source sentence.

We run two different human evaluations in this paper. For our main results, we run a human evaluation based on the Multidimensional Quality Metrics (MQM) methodology (Uszkoreit and Lommel, 2013) with professional translators. Freitag et al. (2021a) showed that this human evaluation is more reliable than typical scalar-value evaluation using crowd-workers. For ablation studies, we use a scalar-value human evaluation with professional translators similar to what is typically implemented in WMT as this human evaluation setup is cheaper and less time-consuming.

We hired 9 professional translators (4 for En→De and 5 for De→En) and measure translation quality with a document context version of MQM (Lommel et al., 2014) which mimics the setup proposed in Freitag et al. (2021a). This includes using the same error categories, severity levels, and error weighting schema. As suggested in the study, we weight each major error with 5 and each minor error with 1, except for minor punctuation errors, which get a score of 0.1. The final segment-level score is an average over scores from all annotators. We refer the reader to Freitag et al. (2021a) for the details on error categories and annotator instructions.

In some of our ablation experiments, we conduct a human evaluation via profesional Scalar Quality Metric (Freitag et al., 2021a). This evaluation presents each source and translated segment from a document in a table row, asking professional translators to pick a rating from 0 through 6. The rater can scroll up or down to see all the other source/translation segments from the document. The final score for each of the systems is an average over their segment-level scores. We run pSQM evaluations in our ablation studies for En→De with 3 professional translators.

In this section, we discuss the main results of our study. First, we look into the automatic scores to investigate if MBR results in higher actual utility scores when estimating the expectation of the same utility. Second, we look into the human evaluation results to investigate how well the improvements in utility scores can transfer to human judgments.

MBR decoding chooses the translations with the highest estimated utility in a candidate list with the hope that this translation also gets a high actual utility score with respect to a human reference. We run MBR decoding with the utilities sBleu, Chrf, Yisi, Bleurt v0.1, and Bleurt v0.2. We verify whether our NMT model is accurate enough for its candidate list to serve as a proxy for the human distribution. Experimental results with a 1,000 candidate list generated by unbiased sampling are summarized in Tables 2 and 3. For all utilities, the hypotheses with the highest estimated utility can generate a higher actual utility (bold, underlined numbers) when compared to the beam search output. This shows that the expectation of utility under the model distribution is a good proxy for the actual utility with respect to a human translation.

Interestingly, MBR with overlap-based metrics (sBleu, Chrf, Yisi) prefers high log likelihood hypotheses, with logP similar to MAP decodes. Rewarding reference overlap—even with an embedding distance in the case of Yisi—favors the most common wording with the highest chance to match the surface form or embedding of a phrase in the reference translation. The Bleurt metrics, on the other hand, do not rely on overlap evaluation and can reward less frequent translations. Bleurt selects alternative translations, which are not scored highly by overlap metrics like Bleu and which are not among the highest likelihood (logP) sentences according to the underlying NMT model.

Automatic metric results are encouraging but need to be confirmed with human assessments. We ran MQM-based human evaluations with professional translators for all MBR decoding outputs, beam search, and one human translation. MQM generates an interpretable error score (lower is better) and a score of 1 is equivalent to an average of one minor error per sentence, while a score of 5 is equivalent to an average of 1 major error. The MQM results in Tables 2 and 3 show that MBR decoding with Bleurt clearly outperforms (significantly, in the case of En→De) beam search decoding and MBR decoding with sBleu, Chrf and Yisi, demonstrating that when comparing different decoding strategies, model probability and actual human assessment poorly correlate. Interestingly, MBR using Bleu as the utility function is also better than beam search decoding, while Chrf and Yisi are ranked below beam search for at least one language pair.

We have to mention that the human translation for En→De outperforms all machine generated translations. For De→En, the human translation is ranked behind all machine generated translations. We looked into the ratings and confirm that the human translation contains critical errors (this is in line with the official WMT21 human evaluation [Barrault et al., 2021]), showcasing how important it is to generate a good human translation when comparing MT with humans.

We run ablation experiments to better understand the properties of MBR. We will mostly focus on experiments for English→German due to space and cost constraints.

The candidate lists used by MBR in the main results section (Section 5) were generated by an NMT model using 375 million parameters similar to the transformer-big architecture. We raise the question if MBR using Bleurt v0.2 still avoids Pitfall 1 and outperforms beam search when using a candidate list that is generated by a weaker model that is trained with 93 million parameters (model dimension size of 512, hidden dimension size of 2,048, and 8 transformer heads) similar to the transformer-base architecture. Experimental results can be seen in Table 4. We can see that the performance drops by 2 Bleu and 2 Bleurt points when comparing the beam hypotheses of the two different NMT models, indicating that the smaller model is indeed of lower quality.

Even though MBR outperforms beam decoding by 0.8 Bleurt points on the transformer-base model, the gap is much smaller than what we observe with the bigger model (7.4 Bleurt points). This already indicates that MBR is less effective on the smaller model, and the candidate list might not be good enough as a proxy for human references. We run a human evaluation comparing the two decoding algorithms on the small model and find that translation quality actually drops for the small setup when using MBR decoding. This shows that MBR requires a good quality candidate list to outperform beam search.

MBR uses the candidate list in two ways: (i) as a candidate pool from which it picks the hypothesis with the maximum estimated Bayes risk (max step) and (ii) as a list of pseudo-references to calculate the expected risk for each entry in the candidate pool ($E$ step). It is not required that both operations use the same list. We run MBR decode using the candidate list of the small model on the $E$ side and the candidate list of the larger model on the max side and vice versa. The Bleurt v0.2 results in the last two rows of Table 4 show that the candidate list generated by the smaller model has a larger negative effect when used on the max operation compared to using it on the $E$ side only. Overall, the results show that it is not sufficient to use the candidate list of the smaller model on either the $E$ or the max operation.

All our MBR decoding results in the main results section (Section 5) rely on a candidate list size of 1,000. Generating 1,000 candidates and computing 1,000× 1,000=1M Bleurt scores for each source sentence is computationally costly and would not be practical at scale. We explore two different strategies to prune the candidate list via either (i) random sampling, or (ii) based on the model probabilities (logP). Similar to Section 6.1, we can apply the pruning strategies to either the $E$ list, the max list or both lists. Experimental results can be seen in Figure 1.

There are three major insights: (i) if we prune both operations in MBR, randomly down- sampling the candidate list size to a size of 8 (En→De) or 13 (De→En) already outperforms beam decoding based on Bleurt. (ii) We can aggressively sub-sample the candidate list used for the expectation (⁠$E$⁠). For En→De, we observe major improvements over beam search decoding, shrinking the candidate list to 5 on the $E$ side, resulting in only 5× 1,000=5,000 Bleurt computations for a single source sentence. This confirms the findings of Section 6.1 that we rely more on the quality and size of the candidate pool on the maximization step than on the expectation. (iii) The results in Figure 1 suggest that the MBR output most likely further improves when increasing the candidate list size beyond 1,000. This is different from beam search where accuracy gains are typically not achieved by growing beam size beyond a small number (<10).

We conduct oracle experiments to evaluate how the MBR hypotheses compare with selecting the best hypothesis with respect to a human reference. Given a human translation ref_human, we select the best hypothesis according to $maxh∈Hmodel$ Bleurt(h, ref_human) and report its Bleurt score. This assesses the gap between our decoding strategy and an oracle decision.

We consider two scenarios: selecting and evaluating a hypothesis with the same human reference, or selecting a hypothesis with a first reference before evaluating it with a second, different reference. The second method considers the selection reference and the evaluation reference as two independent samples of the human translation space. This avoids biasing selection to translation choices specific to the evaluation conditions.

Table 5 reports these results. In the different reference scenario, MBR performs better than the cross human selection, for example, selecting the best hypotheses with Ref-C yields a Bleurt score of 0.774 with Ref-D which is lower than 0.789, the Bleurt score of MBR with Ref-D. It is remarkable that the inter-translator variability in single reference automated evaluation causes more damage in oracle selection than the drop due to swapping human references for model estimates.

Table 6 shows percentiles of the rankings of the selected translations among the candidate list as ranked by Bleurt v0.2 with respect to Ref-C. The median ranking (p50) of the MBR output is 8 out of 1,000, while the median raning of the MAP hypothesis is only 181. Interestingly, the MBR output even achieved higher ranking than the oracle candidate selected by Ref-D Bleurt v0.2 score, confirming the observation in Table 5 that model-estimated MBR provides more reliable quality estimates than selecting hypothesis with a single human reference translation.

Similar to reference-based metrics, reference-free –Quality Estimation (QE)–metrics have made huge improvements in the last years and show promising performance for some language pairs and test sets (Mathur et al., 2020). We pose the question whether a QE metric alone is sufficient to rerank the candidate list that we usually use for MBR decoding. The obvious advantage is that we only need N (N being the size of the candidate list), instead of N × N metric calculations. We present results with two different QE metrics: COMET-QE-20 (Rei et al., 2020) and COMET-QE-21 (Rei et al., 2021). These two metrics were the best QE metrics based on the two most recent WMT metric tasks (Mathur et al., 2020; Freitag et al., 2021b). Experimental results for En→De and De→En can be seen in Table 7.

Both reranking experiments show similar patterns: The QE-based reranking outputs outperform beam search and MBR with Bleurt v0.2 on both QE-metrics. Nevertheless, we can see that most reference-based metrics set the QE-based reranked output below both the beam search and the MBR output. When looking into the translations, we observed that some sentences in the QE-based reranking approach contain translations with crucial errors or the translation is unrelated to the source sentence. The human evaluation results in Table 7 confirm our impression that the reranked translations are of lower quality when compared to our MBR output or the beam search hypothesis. One potential reason of the underperforming reranking experiments can be the quality of the candidate list. As a reminder, the candidate list consists of unbiased samples drawn from the NMT model. Some of the samples are of bad quality and partially or entirely unrelated to the source sentence. While MBR compares the different samples with each other and penalized samples that are different to the other ones, the reranking approach solely relies on the QE metrics and does not have this safety mechanism.

In Section 5, we observed that the model probabilities of the MBR output using Bleurt v0.2 is lower when compared to the beam search output. We want to further characterize the differences between these two decoding algorithms.

Bleu measures the lexical overlap between a hypothesis and a reference translation. It can also be used to measure the lexical similarity of two alternative machine translations. In that case, Bleu does not assess translation quality but surface proximity between sentences.

Cross Bleu scores of our MBR outputs with our MAP decode and the best submissions in WMT21 can be seen in Table 8. Bleu scores lower than 50 are highlighted in the table. Our MAP hypothesis, the WMT21 submissions, and our MBR hypotheses using Bleu, Chrf, or Yisi have high crossBleu, which shows that they yield similar translations. The MBR output using Bleurt and the human translations have low cross-BLEU with all MAP hypotheses, which means that they use different words and sentence structures. It is worth highlighting that the two human translations are as different from each other as they are to our MBR output using Bleurt.

In addition to an overall quality score, MQM provides individual error labels with category and severity information. Table 9 reports major error counts for the most frequent categories, excluding categories with similar counts from beam and MBR. This table shows a clear advantage for the MBR output for four categories. Specifically, the number of errors in the category Terminology/Inappropriate for context which is problematic for En→De shows a reduction of one third with MBR.

We explored an alternative to the commonly used beam search decoding algorithm typically used in NMT. We run the sampling-based approximation of Minimum Bayes Risk (MBR) decoding to optimize Bleu, Chrf, Yisi, and Bleurt. Our experimental results showed that MBR decoding using Bleurt as utility function results in translations that significantly outperform beam search decoding based on expert-based human evaluation. We showed that the resulting translations are significantly different from both the beam search decode and MBR decoding output using one of the other overlap-based metrics as utility function, and have a lower model probability.

We would like to thank Wolfgang Macherey, George Foster, Thibault Sellam, Macduff Hughes, and Orhan Firat for insightful discussions and reviewing the paper. The authors would also like to thank the anonymous reviewers and the action editor of TACL for their constructive reviews.