An Energy-based Model for Word-level AutoCompletion in Computer-aided Translation

Computer-aided Translation (CAT) (Barrachina et al., 2009; Santy et al., 2019; Huang et al., 2021), which enables the leveraging of machine translation systems (Bahdanau et al., 2015; Vaswani et al., 2017) to improve the efficiency of the human translation process, has seen increasing interest in recent years. In this work, we study a crucial yet challenging task in CAT: Word-Level AutoCompletion (WLAC) (Li et al., 2021), which aims at yielding word-level suggestions based on context pieces provided by human (Figure 1(a)).

Previous research includes statistical methods (Huang et al., 2015) and neural methods (Santy et al., 2019; Li et al., 2021). With the help of word alignment toolkits (Och and Ney, 2003; Dyer et al., 2013), statistical approaches build a translation table and use it to predict the target word. More recently, Li et al. (2021) use a Transformer-based classification model, which firstly encodes the input context to a hidden vector and then maps the hidden vector into the candidate target word through a linear classifier. This strong baseline method achieves the state-of-the-art (SOTA) performance.

In the aforementioned classification paradigm, the hidden vector of the input context inherently does not take the candidate target word into consideration. As a result, it may not effectively leverage valuable information carried by the candidate target word when occurring in the input context, as shown in Figure 1(b). Specifically, given the input context and human typed characters “d”, the user may tend to type “disease” (“Krankheit” in German). However, through visualizing attention weights, it shows that the baseline method captures more information from “gemeinsame” and “verzweifelten” than that from the most informative word “Krankheit” in the source side, which may underestimate the model score of the ground-truth word “disease” and thereby leads to incorrect prediction.

To alleviate the above issue, we formalize the WLAC task with an energy-based model (Ranzato et al., 2006; LeCun et al., 2006) based on Transformer, where the hidden vector is defined on top of both the candidate target word and the input context through a deep energy function. Furthermore, with the help of deep neural networks, the energy-based function is expected to capture sufficient information for each candidate target word through the attention mechanism. In this way, the energy function is able to capture informative context (i.e., “Krankheit”) to evaluate the target word (i.e., “disease”), and thereby the score from the energy-based model is more reliable, as shown in Figure 1(c).

Unfortunately, training and inference with the energy-based model suffer from efficiency and effectiveness challenges due to the normalization term in the model. To alleviate the effect of these barriers, we systematically incorporate three simple yet effective strategies inspired by previous studies: (1) a negative sampling method for efficient training (Ma and Collins, 2018; Li et al., 2019a; Xu et al., 2022), (2) a reranking paradigm as an approximate proxy for efficient inference (Shen et al., 2004; Nogueira and Cho, 2019; Bhattacharyya et al., 2021), and (3) a pre-training method for effective training (Lee et al., 2021a). Experiments on four standard benchmarks demonstrate that the energy-based model is indeed better at capturing informative signals for the prediction of a candidate target word and thereby yields substantial improvements over strong baselines.

To sum up, our contribution is three-fold:

1. We point out that the previous SOTA model for the WLAC task suffers from an issue, i.e., it can not sufficiently leverage the valuable information from the source sentence for word prediction.

2. We propose an energy-based model to alleviate this issue and we employ three simple yet effective strategies to put it into practice.

3. We comprehensively evaluate our approach on four benchmarks, and our approach achieves substantial improvements (about 6.07%) over the previous SOTA model.

In this section, we review the setting of the WLAC task and introduce the state-of-the-art baseline method, which will be reused in Section 3.

Let x = (x₁, x₂,…, x_T) be a source sentence, s = (s₁, s₂,…, s_k) be a sequence of human typed characters and c = (c_l, c_r) be translation context where c_l = (c_l,1, c_l,2,…, c_{l, m}) and c_r = (c_r,1, c_r,2,…, c_{r, n}). c_l and c_r are on the left- and right-hand side of s, respectively. Figure 1(a) illustrates the examples for x, c_l, c_r, and s.

Given the input tuple (x, c, s), the WLAC task aims at predicting the target word w, which starts with s and is the most appropriate to be placed between c_l and c_r (Li et al., 2021). In partial translation consisting of c_l, w, and c_r, w is not necessary to be consecutive to c_{l, m} and c_r,1. Figure 1(a) gives an illustrative example. To be more general in real-world scenarios, the WLAC task further assumes that c_l and c_r can be empty, which leads to following four translation context types:

• Zero-context: both c_l and c_r are empty;

• Prefix: c_r is empty;

• Suffix: c_l is empty;

• Bi-context: both c_l and c_r are not empty.

It is noteworthy that context types described above are general and encompass context of several conventional translation scenarios, such as prefix-decoding for left-to-right interactive machine translation (IMT) (Knowles and Koehn, 2016) and post-editing (Lee et al., 2021b; Yang et al., 2022). To elaborate, in prefix-decoding, the context falls into the special case of prefix, where c_r is empty and c_l is consecutive to w. In post-editing, the context corresponds to the special case of bi-context, where both c_l and c_r are consecutive to w.

Li et al. (2021) cast WLAC as a word prediction task. Generally, they decompose the WLAC task into two steps: (1) Model the distribution of the target word w using x and c via a Word Prediction Model (WPM); (2) Predict the most appropriate word $ŵ$ which starts with s according to the conditional distribution. Their method achieves state-of-the-art performance.

As shown in Equation (2) in Section 2.2, the baseline WPM essentially maps the hidden vector of the input context (i.e., h_[MASK]) into the candidate target word to predict the most appropriate target word for [MASK]. Furthermore, according to the model architecture of the baseline WPM, the context hidden vector h_[MASK] does not take the candidate target word into consideration (Liu et al., 2016; Li et al., 2018). Therefore, it might be difficult for h_[MASK] to make full use of sufficient information from the source side for accurately predicting the ground-truth target word. Intuitively, the above issue for the baseline WPM in Equation (1) can be demonstrated from the example in Figure 1(b), where we use attention weights to visualize source words which are mostly used in h_[MASK].² From this figure, we see that h_[MASK] uses more information from “gemeinsame” and “verzweifelten” than that from “Krankheit”. Therefore, such a model may underestimate the score for the ground-truth word “disease”, which aligns to “Krankheit” on the source side. Consequently, the baseline WPM may not successfully predict the ground-truth word, leading to sub-optimal performance.

In response to the above issue, this paper proposes an energy-based model which enables defining the hidden vector on top of both the candidate target word and the input context through an energy function. Our intuition is that with the help of deep neural networks (e.g., attention networks), the energy function is expected to capture more valuable information from the source sentence, which makes the model score more reliable to evaluate contributions for w.

The energy-based model in Equation (3) is very general, because the energy function S(w, x, c) can be any function. For example, as a special case, if we set S(w, x, c) = P_b(w|x, c), the energy-based model is then reduced to Equation (1) because the normalization term is 1. Since this paper aims to alleviate the insufficient usage of source sentence information for P_b, it seeks another definition of the energy function to define the hidden vector on top of both the candidate target word w and the input context (x, c).

Theoretically, there are many ways to define the energy function S(w, x, c). In this paper, in practice, we adopt the way to define S(w, x, c) very similar to P_b in model architecture with minimal modifications and almost the same number of parameters as P_b. As a result, it could indicate that the potential improvement derived from the energy-based model is not significantly attributed to the complex model architecture of S(w, x, c), but rather to define the hidden vector on top of both the candidate target word w and the input context (x, c).

We believe that the energy function S can adequately exploit contextual information from (x, c). This belief is exemplified in Figure 1(c).³ In this figure, after visualizing attention weights to source words, the energy function S is able to capture more information from “Krankheit” to evaluate the target word “disease”. Therefore S(disease, x, c) is more reliable than baseline score P_b(disease|x, c), which inadequately make use of the signal from “Krankheit” as shown in Figure 1(b).

However, it is far from trivial to make the energy-based model achieve the effect as shown in Figure 1(c) and further deliver excellent performance on the WLAC task due to the following efficiency and effectiveness challenges.

The first challenge is the efficiency in both training and inference. During training, maximizing the log-likelihood for Equation (3) needs the calculation of the value of the normalization term. During inference, it needs to enumerate all candidate words from vocabulary $V$⁠. Unfortunately, the energy function S sacrifices the parallel computation for all $w∈V$⁠: One has to feed all candidate target words to the network architecture independently for each w. However, since $V$ is too large, such exhaustive computation is infeasible in practice. Consequently, this makes both training and inference challenging for the energy-based model.

Second, in our preliminary experiments, optimizing the energy-based model from scratch does not work well, and its final performance is significantly worse than the baseline P_b. One possible reason is that it is more difficult to train the energy-based model. Training the energy-based model involves an approximate method to shrink the subset for the normalization term, and this may induce a risk that the informative negative examples are excluded in the shrunk subset (Ma and Collins, 2018; Xu et al., 2022). Therefore, it is easy to get trapped in local optimization when training the energy-based model from scratch.

To relieve the aforementioned challenges, we systematically employ three simple yet effective methods inspired by previous studies. First, we employ negative sampling to address the normalization computation during the training (Ma and Collins, 2018; Li et al., 2019a; Xu et al., 2022); similarly, during the inference, we adopt a reranking paradigm, where the energy-based model is used as a reranker over a small subset of candidates (Shen et al., 2004; Nogueira and Cho, 2019; Bhattacharyya et al., 2021). Moreover, we harness a conditional mask bilingual language modeling pre-training strategy for parameter initialization (Lee et al., 2021a).

In this paper, we try different settings for $P^$⁠. As the first setting, $P^$ is defined by the uniform distribution over $V$⁠. Although sampling from this distribution is efficient and even does not introduce extra computation, it cannot ensure the hard negatives are sampled with a high probability. Thus it is not promising to speed up the convergence in our experiments. Hence, as the second setting, $P^$ is instantiated by the baseline model P_b. Furthermore, according to our empirical results, it will achieve better performance by replacing the sampling operation in Equation (4) with the top-K operation over the distribution P_b(w|x, c).

As described before, due to the definition of the energy function S(w, x, c), it is too costly to evaluate S(w, x, c) for all w. Thus, it is infeasible to exactly predict the best w such that S(w, x, c) is maximal. Similar to the top-K operation in the training stage, we adopt it in the inference stage as an approximation. Specifically, the inference process by the energy-based model includes the following two steps:

• Obtain the top-K subset denoted by Ω(s, K) according to P_b(w|x, c), where each element also satisfies the constraint s:

  $Ω(s,K)=TOPw∈V(s)KPb(w∣x,c)$
• Output the target word $ŵ$ in terms of the energy function as follows:

  $ŵ=argmaxw∈Ω(s,K)S(w,x,c)$

  (5)

Recently, pre-trained language models have made exceptional success in numerous natural language processing tasks (Devlin et al., 2019; Lewis et al., 2020; Ouyang et al., 2022). One of their advantages is that they can learn general and contextual representations to boost the downstream tasks (Li et al., 2022, 2023a; Shi et al., 2023). Inspired by this, we propose to use our limited supervised bilingual data to conduct a small-scale pre-training for the energy-based model to yield better weight initialization.

In this section, we first describe the experimental setup. Then we report the main results and analyze the proposed approach.

We experiment on four language pairs: Zh⇒En, En⇒Zh, De⇒En and En⇒De. For training on Zh⇒En and En⇒Zh, we use the training set from the LDC corpus,⁴ which consists of 1.25M sentence pairs. For training on De⇒En and En⇒De, we use the preprocessed WMT14 dataset by Stanford,⁵ which consists of 4.5M sentence pairs. We use the standard validation and test sets released by Li et al. (2021).⁶ Specifically, for Zh⇒En and En⇒Zh, they construct validation set from NIST02 and test set from NIST05 and NIST06. For De⇒En and En⇒De, they extract validation set from newstest13 and test set from newstest14.

In order to construct simulated training data, we follow the same strategy as Li et al. (2021) to sample target words, human typed characters and translation context, which aims at avoiding sampling trivial instances. Statistics of the average length of target words and human typed characters on validation sets are shown in Table 1. As we can see, in general, target words are long and human typed characters are short, which poses a challenge for the WLAC task. In addition, we also conduct a frequency analysis of each word in training set across four language pairs. Following this, words are categorized into ten intervals based on their frequency. Finally, we calculate the proportion of target words in validation sets corresponding to each frequency interval. The result is presented in Figure 3. Figure 3 indicates a non-uniform distribution of target words across different frequency intervals. This data composition basically reflects demands encountered in real-world scenarios, where non-high frequency words are more challenging for WLAC models.

We compare our model with the following baseline models:

• TransTable: A statistical method inspired by Huang et al. (2015). They create a word-level translation table with a word alignment toolkit.⁷ During the inference stage, they use the translation table to obtain translations of all source words and filter out invalid candidate words through human typed characters. Ultimately, they pick the candidate word with the highest frequency as the prediction.

• Trans-PE: A Transformer-based baseline inspired by Langlais et al. (2000) and Santy et al. (2019). They first train a vanilla Transformer on training set. While testing, they only feed the left translation context to the Transformer decoder. Then they conduct a next-word prediction task with human typed characters as hard constraints to get the prediction word.

• Trans-NPE: The only difference between this method between Trans-PE is that there is no position encoding layer in the decoder of Trans-NPE. They apply average pooling to the representations of all translation context words. And then, they use the pooled representation to predict the target word.

• P_b: The word prediction model defined in Equation (1), which is the state-of-the-art model of the WLAC task.

• Trans-BPE: Inspired by De Cao et al. (2021); Yang et al. (2022), we also implement a new Transformer-based baseline over subwords. Specifically, we apply BPE to segment words into subwords. During the inference stage, we adopt Prefix-Constrained Beam Search (De Cao et al., 2021) to generate outputs which start with human typed characters. This model is expected to be capable of defining the hidden vector on top of previously generated subwords and the input context to predict the next subword.

We implement our energy-based model on top of the Transformer-Base architecture (Vaswani et al., 2017) implemented in Fairseq toolkit (Ott et al., 2019).⁸ The source encoder is a stack of 6 Transformer encoder blocks. The target encoder is also composed of 6 blocks, each of which is a Transformer encoder block with an additional cross-attention layer between the multi-head self-attention layer and feed-forward layer. The vocabulary size is 60K for Chinese, 50K for German, and 50K for English. As for the implementation of Trans-BPE, we adopt the Transformer-Base architecture and make adjustments to the input of Transformer Encoder. Specifically, we feed the concatenation of the source context, target context, and placeholder [MASK] to the Transformer Encoder, and adopt segment embedding to distinguish different languages as Yang et al. (2022). The vocabulary size is 32K for both Zh⇔En and De⇔En. For a fair comparison, we also re-implement P_b with the same hyperparameter settings as the energy-based model.

For the above models, we set d_model = 512, d_hidden = 2048, n_head = 8 and p_dropout = 0.1. The learning rate is set as 0.0005, and the warmup step is set as 4,000 steps. All models are trained with 4096 tokens per batch for a maximum of 50,000 steps with the Adam optimizer (Kingma and Ba, 2015) on 8 NVIDIA V100 GPUs. We update the model parameters after accumulating 2 gradients for Trans-BPE and 1 gradient for P_b and Ours. Models are selected with the best accuracy on the validation set. We repeat the main experiment 5 times by using different random seeds.

Table 2 lists the main results on four language pairs. From the table, we can make three observations: First, statistical and intuitive Transformer-based methods (#1-3) perform poorly on all language pairs. We speculate that this is because these approaches can not make full use of the information from the input context (e.g., source sentence). Second, Trans-BPE outperforms P_b on average accuracy. The reason behind this could be attributed to the effectiveness of Trans-BPE to leveraging more valuable source sentence information than P_b, which we will elaborate on in Section 5.4. Third, our energy-based model (#7) improves over the previous SOTA performance by an average of 6.07 accuracy points across all language pairs, which demonstrates its effectiveness. Furthermore, in Table 3 and Table 4, we report the detailed results of different systems on four translation context types on the Zh⇔En and De⇔En validation sets. We can find that our energy-based model can almost achieve performance improvement on each translation context type, except for De⇒En prefix context, and finally results in overall performance in Table 2.

It is also crucial to assess the actual improvement in effectiveness of our approach via human evaluation. However, performing comprehensive human evaluations can be resource-intensive in terms of labor. As a compromise, we randomly sample 400 examples from the original Zh⇒En and En⇒Zh NIST05 test sets, with 100 instances for each translation context type. We then collect predictions from three models: P_b, TransBPE, and Ours. Subsequently, we enlist two professional evaluators to assess the appropriateness of predictions of these models. The human evaluators are presented with the input context, human typed characters, as well as each prediction. The predictions, originating from different models, are anonymized to the evaluators. The human evaluators are asked to assign binary scores for each prediction, where a score of ‘1’ indicates appropriateness, while ‘0’ signifies inappropriateness. Results of human evaluation are presented in Table 5. The Cohen’s kappa is 0.92 between the two translators, which is a relatively high agreement. Table 5 demonstrates that our energy-based model retains an advantage over previous methods under human evaluation. What’s more, one detail worth noting is that, compared to results in Table 2, all models exhibit an improvement in performance when evaluated manually. This can be attributed to the fact that the accuracy metric only considers the top-1 prediction, while other predictions may also be valid. To ensure consistency with prior research, we utilize accuracy as the evaluation metric in the following sections.

As we state in Section 3, negative sampling in the training stage can affect the performance of the energy-based model. We consider two sampling distributions (the uniform distribution and the distribution of P_b) and three negative sampling strategies, i.e., random sampling, top-p sampling and top-K sampling. We compare them on Zh⇒En dataset. During the inference stage, we use P_b to recall top-8 predicted words as candidate target words for these models trained with different negative sampling techniques.

We report the results in Table 7. We can observe that the random sampling strategy from the uniform distribution is not as effective as the other three sampling configurations from P_b. We conjecture that negative samples by random sampling on the uniform distribution could be too trivial to recognize hard negatives, which may hinder the performance of the energy-based model. While sampling according to P_b (i.e., the other three strategies) can sample hard negatives and facilitate the training of the energy-based model.

We further analyze the impact of candidate word set size $K=V(s)$ during the inference with the energy-based model. Figure 4 shows that, as K increases, the accuracy improvement increases rapidly from K = 1 to K = 4 and starts to saturate after K = 4. The recall of the ground-truth word shares the same trend as accuracy: It first improves sharply, then increases slowly and reaches a relatively high value. So for the efficiency and effectiveness trade-off, we choose to use K = 8 as our candidate word set size in all experiments during the inference.

Our energy-based model is pre-trained by a CMBLM pre-training strategy. Therefore, its improvements might come from two aspects, including 1) the energy-based model and 2) better initialization weights and representations learned from the CMBLM pre-training task. Hence, we perform further studies to quantify the contribution of each component of our approach. To this end, we conduct two experiments: we replace the CMBLM pre-training by initializing the weights from the baseline WPM P_b; and we apply the CMBLM pre-training on top of P_b and compare it with the energy-based model with the CMBLM pre-training. We evaluate all these methods on Zh⇒En dataset and De⇒En dataset and present the results in Table 6.

The results in Table 6 illustrate that: First, initializing the weights of the energy-based model with P_b is not as effective as initializing with the CMBLM pre-training strategy. Second, although both P_b and our energy-based model benefit from the CMBLM pre-training strategy, the gain for the energy-based model is much larger. These observations demonstrate that a simple pre-training method can not activate the potential of the energy-based model and the CMBLM pre-training strategy succeeds.

Although our work mainly focuses on four translation context types in the WLAC task, we also explore whether the energy-based model would still improve performance on two common translation scenarios including prefix-decoding widely used in left-to-right interactive machine translation and post-editing as stated in Section 2.1. To this end, we implement P_b, Trans-BPE and Ours on these two scenarios with the same parameter configuration in Section 5.1. As for the construction of validation sets and test sets, we adopt the same simulation method as Li et al. (2021) other than that the target word must be consecutive to target context. Table 8 shows the results of P_b, Trans-BPE, and Ours on prefix-decoding and post-editing scenarios. As we can see, Ours can further improve average accuracy points across all language pairs by 3.22 on post-decoding and by 2.68 on post-editing, demonstrating the effectiveness of our energy-based model.

As we have claimed in Section 3, our motivation is that the energy-based model is capable of capturing more informative context for word prediction, which thereby leads to better performance eventually. In addition to the intuitive example in Figure 1(c), we design an automatic metric to verify our motivation. This metric is inspired by the word alignment error rate for the cross-attention in the Transformer (Li et al., 2019b; Garg et al., 2019). Specifically, as shown in Figure 1(c), the metric (alignment recall@n) is defined as the recall rate of the informative source word “Krankhof Type-II errors and eit” by the top-n source words according to the attention score by the Transformer architecture. For each ground-truth target word, e.g., “disease” in Figure 1(c), the informative source word is defined by the manually annotated word alignment.

We use the human-annotated alignment data on Zh⇔En NIST05 dataset and conduct experiments in the bi-context scenario. We compare the alignment recall@n between P_b, Trans-BPE and Ours in Figure 5. As we can see, the alignment recall@1 of Ours is higher than P_b by 60 points and when n is small, it always maintains this advantage. What’s more, Trans-BPE also achieves better alignment recall@n than P_b. This may serve as quantitative evidence that introducing subwords or the entire candidate target word into the modeling of hidden vectors with the input context, as implemented in Trans-BPE and Ours, can make more use of informative context than P_b (De Cao et al., 2021). And results illustrated in the Figure 5 also reveal that our energy-based model might be more effective in leveraging informative context than Trans-BPE.

After conducting the human evaluation in Section 5.2, we proceed to inspect incorrect instances of P_b and Ours in Zh⇒En test examples.

Furthermore, we summarize incorrect instances into three distinct categories: (1) Semantic discrepancy error (Type-I): The model erroneously suggests irrelevant words. These words lack semantic relevance to source sentences other than starting with the same human typed characters. (2) Repetition error (Type-II): The model suggests words that convey semantics of source sentences, however, these words already appear within the target context. (3) Morphological error (Type-III): The model suggests incorrect cognates of target words.⁹ In the forthcoming Case Study section, we will present illustrative examples representing each of these three error categories.

In Table 9, we present quantitative results of error occurrences for P_b and Ours. In terms of the total error quantity, Ours exhibits a lower number of errors. Notably, for both methods, the most common error type is semantic discrepancy error. Comparatively, Ours demonstrates a notable ability to rectify 25 instances (31.65%) of Type-I errors, 20 instances (68.97%) of Type-II errors, and 14 instances (70.00%) of Type-III errors that are present in P_b. Furthermore, Ours exhibits significantly fewer instances in repetition and morphological errors. However, it is essential to acknowledge that the Ours approach also introduces new incorrect instances in each type that are not originally observed in P_b.

We provide this case study to better illustrate the advantages of Ours over P_b in utilizing contextual information, thereby leading to enhanced semantic information for word-level autocompletion. Figure 6 presents cases where P_b yields errors while Ours predicts correctly. Furthermore, Figure 7 illustrates their attention weights which depict the connection between the predicted word and the source words.

In case 1 (Type-I), P_b tends to suggest “suffice”, which is not consistent with semantics expressed by the source sentence other than starting with human typed characters “suf”. In contrast, Ours succeeds in completing “suf” to “suffer”. Through visualizing attention weights in Figure 7, we can find that Ours may have the merit of leveraging more information from the valuable source context (e.g., the aligned word “饱受”). In case 2 (Type-II), P_b completes “so” to “social”, which has already been translated in target context. With the leverage of interactions between candidate target words and input context, Ours successfully suggests “services”. In case 3 (Type-III), P_b suggests the cognates of target words (i.e. “problematic”). Whereas, according to the information captured in the energy-based model, Ours succeeds in suggesting the noun “problems”, which are more appropriate. Although our model has substantially alleviated aforementioned cases, it is not flawless. One such instance is that, during the inference stage, the effectiveness of Ours is influenced by the baseline recall rate.

Table 10 summarizes the training and inference latency of P_b, Trans-BPE, and Ours on Zh⇒En validation dataset. The results indicate that the training and inference latency of Ours is comparatively higher than that of P_b (approximately 2.0 times and 1.5 times, respectively). This discrepancy in latency can be attributed to the inherent necessity of Ours to get candidate words from P_b and subsequently rerank them, which demands additional computational time. In comparison to the more potent auto-regressive model, Trans-BPE, Ours exhibits a lower inference latency while concurrently delivering better performance. As a result, our approach achieves a desirable balance between performance and processing speed.

As stated in the previous sections, one advantage of WLAC is that it is able to increase the efficiency of human input in interactive machine translation. To exemplify the usefulness of WLAC, we apply the WLAC models into IMT. Specifically, we first implement a practical IMT model following Huang et al. (2021) which is based on lexical constrained decoding (Hokamp and Liu, 2017) and thus enables the flexible input from users. Then, we apply three WLAC models (P_b, Trans-BPE, and Ours) into the IMT model, leading to three IMT systems named by IMT-P_b, IMT-Trans-BPE, and IMT-Ours. As a direct baseline, the IMT system without WLAC is denoted by IMT-Raw.

For efficiency evaluation in IMT, the standard metric, the number of keystrokes from a human translator (Nepveu et al., 2004; Bender et al., 2005), is used for all IMT systems. To ensure a fair comparison in efficiency, we enforce all human inputted words to be the same for all IMT systems and thus all these IMT systems yield the same translation outputs. We randomly select a subset consisting of 200 source sentences from Zh⇒En NIST05 as x due to intensive human efforts in IMT experiments. On this subset, the standard NMT obtains 50.13 BLEU points and all IMT systems achieve 56.02 BLEU points thanks to human interactions.

Table 11 presents the total and average number of keystrokes across different IMT systems. Notably, the employment of WLAC systems significantly reduces the number of keystrokes in comparison to the IMT-Raw baseline without WLAC. Furthermore, in comparison to other systems, our proposed IMT-Ours system attains a minimal number of keystrokes relative to other systems. This observation is reinforced in Figure 8, which depicts the distribution of the number of keystrokes across different systems. We can see that most of the keystrokes of Ours are less than 3 (constituting approximately 84.5% of cases), leading to a reduction in the number of keystrokes and offering input convenience for users.

Computer-aided Translation (CAT) (Langlais et al., 2000; Barrachina et al., 2009; Green et al., 2014; Knowles and Koehn, 2016; Santy et al., 2019; Lee et al., 2021b) has the merit of leveraging advantages of machine translation systems to facilitate human translation process. Word-level AutoCompletion (WLAC) is an important feature of interactive CAT (Casacuberta et al., 2022) and it plays an important role in CAT. Huang et al. (2015) leverage useful source-side knowledge to complete the target word. Li et al. (2021) propose a strong word prediction model (WPM) and try to leverage both source-side and target-side information. However, as stated in Section 1, these methods may still inadequately leverage the valuable information from the source sentence. To fill this gap, we introduce an energy-based model to enable the hidden vector to capture more valuable information.

Reranking has been long researched in natural language processing tasks (Shen et al., 2004; Collins and Koo, 2005; Charniak and Johnson, 2005). Recently, the retrieval-then-reranking framework has served as the de facto paradigm (Nogueira and Cho, 2019; Zhang et al., 2022) in text retrieval. To yield high-quality answers, answer reranking is also widely employed in question answering (Wang et al., 2018; Iyer et al., 2021), dialogue systems (Li et al., 2023b), and reasoning (Kazemi et al., 2023; Zhu et al., 2023a, b). In machine translation, with the purpose of alleviating the mismatch between maximum likelihood estimation and the desired metric (e.g., BLEU), Bhattacharyya et al. (2021) and Lee et al. (2021a) propose to train an energy-based model to rerank candidate translations generated by NMT models. In this work, we are in line with prior findings that reranking is a conceptually simple yet empirically powerful framework. However, we pay more attention to leveraging valuable source sentence information in the WLAC task and corresponding training and inference challenges of the energy-based model for reranking.

In recent years, with the advance of neural networks, the input method has shown significant progress in being effective (Huang et al., 2018; Zhang et al., 2019; Tan et al., 2022). However, most current research has concentrated on the monolingual scenarios, without sufficient consideration of how to utilize source-side information in bilingual settings (Li, 2012; Huang et al., 2015). Our work, which centers on the word-level autocompletion task to reduce keystrokes, is a new exploration of bilingual input methods. We believe that combining our approach with other input method technologies could significantly enhance the productivity of human translators. We leave this as a potential direction for future research.

Word-level AutoCompletion is a critical yet challenging task in Computer-aided Translation. Existing work casts this task as a classification problem. However, it cannot make full use of the contextual information from the input context for its prediction. To alleviate such issue, we introduce a reranking perspective by an energy-based model, which directly defines the energy function on top of the input context and the candidate target word. Extensive experiments and analyses demonstrate the effectiveness of our proposed approach on four standard benchmarks: It achieves about 6.07% improvements over the strongest baseline.

This work was partly supported by the National Natural Science Foundation of China (grant no. U1903213) and the Shenzhen Science and Technology Program (JSGG20220831093004008). We extend our thanks to annotators for their substantial contributions to this project. Additionally, we would like to convey our appreciation to the TACL editors and anonymous reviewers for their valuable feedback, which significantly enhanced the paper’s quality.