Language Models for Machine Translation: Original vs. Translated Texts

Statistical machine translation (MT) uses large target language models (LMs) to improve the fluency of generated texts, and it is commonly assumed that for constructing language models, “more data is better data” (Brants and Xu 2009). Not all data, however, are created the same. In this work we explore the differences between language models compiled from texts originally written in the target language (O) and language models compiled from translated texts (T).

This work is motivated by much research in Translation Studies that suggests that original texts are significantly different from translated ones in various aspects (Gellerstam 1986). Recently, corpus-based computational analysis corroborated this observation, and (Kurokawa, Goutte, and Isabelle 2009) apply it to statistical machine translation, showing that for an English-to-French MT system, a translation model trained on an English-translated-to-French parallel corpus is better than one trained on French-translated-to-English texts. The main research question we investigate here is whether a language model compiled from translated texts may similarly improve the results of machine translation.

We test this hypothesis on several translation tasks, including translation from several languages to English, and two additional tasks where the target language is not English. For each language pair we build two language models from two types of corpora: texts originally written in the target language, and human translations from the source language into the target language. We show that for each language pair, the latter language model better fits a set of reference translations in terms of perplexity. We also demonstrate that the differences between the two LMs are not biased by content, but rather reflect differences on abstract linguistic features.

Research in Translation Studies holds a dual view on translationese, the sublanguage of translated texts. On the one hand, there is a claim for so-called translation universals, traits of translationese which occur in any translated text irrespective of the source language. Others hold, on the other hand, that each source language “spills over” to the target text, and therefore creates a sub-translationese, the result of a pair-specific encounter between two specific languages. If both these claims are true then language models based on translations from the source language should best fit target language reference sentences, and language models based on translations from other source languages should fit reference sentences to a lesser extent yet outperform originally written texts. To test this hypothesis, we compile additional English LMs, this time using texts translated to English from languages other than the source. Again, we use perplexity to assess the fit of these LMs to reference sets of translated-to-English sentences. We show that these LMs depend on the source language and differ from each other. Whereas they outperform O-based LMs, LMs compiled from texts that were translated from the source language still fit the reference set best.

Finally, we train phrase-based MT systems (Koehn, Och, and Marcu 2003) for each language pair. We use four types of LMs: original; translated from the source language; translated from other languages; and a mixture of translations from several languages. We show that the translated-from-source-language LMs provide a significant improvement in the quality of the translation output over all other LMs, and that the mixture LMs always outperform the original LMs. This improvement persists even when the original LMs are up to ten times larger than the translated ones. In other words, one has to collect ten times more original material in order to reach the same quality as is provided with translated material.

It is important to emphasize that translated texts abound: in fact, Pym and Chrupała (2005) show (quantitatively!) that the rate of translations into a language is inversely proportional to the number of books published in that language: So whereas in English only around 2% of texts published are translations, in languages such as Albanian, Arabic, Danish, Finnish, or Hebrew translated texts constitute between 20% and 25% of the total publications. Furthermore, such data can be automatically identified (see Section 2). The practical impact of our work on MT is therefore potentially dramatic.

The main contributions of this work are thus a computational corroboration of the following hypotheses:

• 1. 

  Original and translated texts exhibit significant, measurable differences.

• 2. 

  LMs compiled from translated texts better fit translated references than LMs compiled from original texts of the same (and much larger) size (and, to a lesser extent, LMs compiled from texts translated from languages other than the source language).

• 3. 

  MT systems that use LMs based on manually translated texts significantly outperform LMs based on originally written texts.

This article¹ is organized as follows: Section 2 provides background and describes related work. We explain our experimental set-up, research methodology and resources in Section 3 and detail our experiments and results in Section 4. Section 5 discusses the results and their implications, and suggests directions for future research.

Numerous studies suggest that translated texts are different from original ones. (Gellerstam 1986) compares texts written originally in Swedish and texts translated from English into Swedish. He notes that the differences between them do not indicate poor translation but rather a statistical phenomenon, which he terms translationese. He focuses mainly on lexical differences, for example, less colloquialism in the translations, or foreign words used in the translations “with new shades of meaning taken from the English lexeme” (page 91). Only later studies consider grammatical differences (see, e.g., Santos 1995). The features of translationese were theoretically organized under the terms laws of translation and translation universals.

(Toury 1980; Toury 1995) distinguishes between two laws: the law of interference and the law of growing standardization. The law of interference pertains to the fingerprints of the source text that are left in the translation product. The law of standardization pertains to the effort to standardize the translation product according to existing norms in the target language (and culture). Interestingly, these two laws are in fact reflected in the architecture of statistical machine translation: Interference in the translation model and standardization in the language model.

The combined effect of these laws creates a hybrid text that partly corresponds to the source text and partly to texts written originally in the target language, but in fact belongs to neither (Frawley 1984). (Baker 1993; Baker 1995; Baker 1996) suggests several candidates for translation universals, which are claimed to appear in any translated text, regardless of the source language. These include simplification, the tendency of translated texts to simplify the language, the message or both; and explicitation, their tendency to spell out implicit utterances that occur in the source text.

During the 1990s, corpora were used extensively to study translationese. For example, (Al-Shabab 1996) shows that translated texts exhibit lower lexical variety (type-to-token ratio) and (Laviosa 1998) shows that their mean sentence length is lower, as is their lexical density (ratio of content to non-content words). These studies, although not conclusive, provide some evidence for the simplification hypothesis.

(Baroni and Bernardini 2006) use machine learning techniques to distinguish between original and translated Italian texts, reporting 86.7% accuracy. They manage to abstract from content and perform the task using only morpho-syntactic cues. (Ilisei et al. 2010) perform the same task for Spanish but enhance it theoretically in order to check the simplification hypothesis. They first use a set of features which seem to capture “general” characteristics of the text (ratio of grammatical words to content words); they then add another set of features, each of which relates to the simplification hypothesis. Finally, they remove each “simplification feature” in turn and evaluate its contribution to the classification task. The most informative features are lexical variety, sentence length, and lexical density.

(van Halteren 2008) focuses on six languages from Europarl (Koehn 2005): Dutch, English, French, German, Italian, and Spanish. For each of these languages, a parallel six-lingual subcorpus is extracted, including an original text and its translations into the other five languages. The task is to identify the source language of translated texts, and the reported results are excellent. This finding is crucial: as (Baker 1996) states, translations do resemble each other; in accordance with the law of interference, however, the study of (van Halteren 2008) suggests that translation from different source languages constitute different sublanguages. As we show in Section 4.2, LMs based on translations from the source language outperform LMs compiled from non-source translations, in terms of both fitness to the reference set and improving MT.

(Kurokawa, Goutte, and Isabelle 2009) show that the direction of translation affects the performance of statistical MT. They train systems to translate between French and English (and vice versa) using a French-translated-to-English parallel corpus, and then an English-translated-to-French one. They find that in translating into French it is better to use the latter parallel corpus, and when translating into English it is better to use the former. Whereas they address the translation model, we focus on the language model in this work. We show that using a language model trained on a text translated from the source language of the MT system does indeed improve the results of the translation.

We investigate the following three hypotheses:

• 1. 

  Translated texts differ from original texts.

• 2. 

  Texts translated from one language differ from texts translated from other languages.

• 3. 

  LMs compiled from manually translated texts are better for MT than LMs compiled from original texts.

We test our hypotheses by considering translations from several languages to English, and from English to German and French. For each language pair we create a reference set comprising several thousands of sentences written originally in the source language and manually translated to the target language. Section 3.4 provides details on the reference sets.

To investigate the first hypothesis, we train two LMs for each language pair, one created from texts originally written in the language (O-based) and the other from texts translated into the target language (T-based). Then, we check which LM better fits the reference set.

For the second hypothesis, we extend the experiment to LMs created from texts translated from other languages. For example, we test how well an LM trained on French-translated-to-English texts fits the German-translated-to-English reference set; and how well an LM trained on German-translated-to-English texts fits the French-translated-to-English reference set.

Finally, for the third hypothesis, we use these LMs for statistical MT (SMT). For each language pair we build several SMT systems. All systems use a translation model extracted from a parallel corpus which is oblivious to the direction of the translation; and one of the above-mentioned LMs. Then, we compare the translation quality of these systems in terms of the Bleu metric (Papineni et al. 2002) (as we show in Section 5.1, other automatic evaluation metrics reveal the same pattern).

In all the experiments, we use SRILM (Stolcke 2002) with interpolated modified Kneser-Ney discounting (Chen 1998) and no cut-off on all n-grams, to train n-gram language models from various corpora. Unless mentioned otherwise, n = 4. We limit language models to a fixed vocabulary and map out-of-vocabulary (OOV) tokens to a unique symbol to better control the OOV rates among various corpora. We experimented with two techniques for setting the vocabulary: Use all words that occur more than once in the evaluation set (see Section 3.4); and use the intersection of all words occurring in all corpora used to train the language model. Both techniques produce very similar results, and for brevity we only report the results achieved with the former technique. In addition, we tried various discounting schemes (e.g., Good-Turing smoothing (Chen 1998)), and also ran experiments with an open vocabulary. The results of all these experiments are consistent with our findings, and therefore we do not elaborate on them here.

Our main corpus is Europarl (Koehn 2005), specifically, portions collected over the years 1996–1999 and 2001–2009. This is a large multilingual corpus, containing sentences translated from several European languages. It is organized as a collection of bilingual corpora rather than as a single multilingual one, however, and it is hard to identify sentences that are translated into several languages.

We therefore treat each bilingual subcorpus in isolation; each such subcorpus contains sentences translated to English from various languages. We rely on the language attribute of the speaker tag to identify the source language of sentences in the English part of the corpus. Because this tag is rarely used with English-language speakers, we also exploit the ID attribute of the speaker tag, which we match against the list of British members of the European parliament.²

We focus on the following languages: German (DE), French (FR), Italian (IT), and Dutch (NL). For each of these languages, L, we consider the L-English Europarl subcorpus. In each subcorpus, we extract chunks of approximately 2.5 million English tokens translated from each of these source languages (T-DE, T-FR, T-IT, and T-NL), as well as sentences written originally in English (O-EN). The mixture corpus (MIX), which is designed to represent “general” translated language, is constructed by randomly selecting sentences translated from any language (excluding original sentences). For English-to-German and English-to-French, we use the German–English and French–English Europarl sub-corpora. We extract German (and French) sentences translated from English, French (or German), Italian, and Dutch, as well as sentences originally written in German (or French).

Table 1 lists the number of sentences, number of tokens, and average sentence length for each English subcorpus and each original language. Table 2 lists the statistics for German and French corpora.

In another set of experiments we address the size of language models, to assess how much more original material is needed compared with translated material (Section 4.2.2). Because Europarl does not have enough training material for this task, we use the Hansard corpus, containing transcripts of the Canadian parliament from 1996–2007. This is a bilingual French–English corpus comprising about 80% original English texts (EO) and about 20% texts translated from French (FO). We first separate original English texts from texts translated from French and then, for each subcorpus, we randomly extract portions of texts of different sizes: 1M, 5M, and 10M tokens from the FO corpus and 1M, 5M, 10M, 25M, 50M, and 100M tokens from the EO corpus; see Table 3. For even larger amounts of data, we use the English Gigaword corpus (Graff and Cieri 2007), from which we randomly extract portions of up to 1G tokens; see Table 4. Unfortunately, we do not know how much of this corpus is original; because it includes data from the Xinhua news agency, we suspect that parts of it are indeed translated.

To experiment with a non-European language (and a different genre) we choose Hebrew (HE). We use two English corpora: The original (O-EN) corpus comprises articles from the International Herald Tribune, downloaded over a period of seven months (from January to July 2009). The articles cover four topics: news (53.4%), business (20.9%), opinion (17.6%), and arts (8.1%). The translated (T-HE) corpus consists of articles collected from the Israeli newspaper HaAretz over the same period of time. HaAretz is published in Hebrew, but portions of it are translated to English. The O-corpus was downsized in order for both subcorpora to have approximately the same number of tokens in each topic. Table 5 lists basic statistics for this corpus.

To focus on the effect of the language model on translation quality, we design SMT training corpora to be oblivious to the direction of translation. Again, we use Europarl (January 2000 to September 2000) as the main source of our parallel corpora. We also use the Hansard corpus: We randomly extract 50,000 sentences from the French-translated-to-English subcorpus and another 50,000 sentences from the original English subcorpus. For Hebrew we use the Hebrew–English parallel corpus (Tsvetkov and Wintner 2010) that contains sentences translated from Hebrew to English (54%) and from English to Hebrew (46%). The English-to-Hebrew part comprises many short sentences (approximately six tokens per sentence) taken from a movie subtitle database. This explains the low average sentence length of this particular corpus. Table 6 lists some details on those corpora.

The reference sets have two uses. First, they are used as the test sets in the experiments that measure the perplexity of the language models. Second, in the MT experiments we use them to randomly extract 1,000 sentences for tuning and 1,000 (different) sentences for evaluation. All references are of course disjoint from the LM and training materials.

For each language L we use the L-English subcorpus of Europarl (over the period of October 2000 to December 2000). For L-to-English translation tasks we only use sentences originally produced in L, and for English-to-L tasks we use sentences originally written in English. The Hansard reference set comprises only French-translated-to-English sentences. The Hebrew-to-English reference set is an independent (disjoint) part of the Hebrew-to-English parallel corpus. This set mostly comprises literary data (88.6%) and a small portion of news (11.4%). All sentences are originally written in Hebrew and are manually translated to English. See Table 7 for the figures.

We detail in this section the experiments performed to test the three hypotheses: that translated texts can be distinguished from original ones, and provide better language models for other translated texts; that texts translated from other languages than the source are still better predictors of translations than original texts (Section 4.1); and that these differences are important for SMT (Section 4.2).

We begin with English as the target language. We train 1-, 2-, 3-, and 4-gram language models for each Europarl subcorpus, based on the corpora described in Section 3.2. For each language L, we compile a LM from texts translated (into English) from L; from texts translated from languages other than L (including a mixture of such languages, MIX); and from texts originally written in English. The LMs are applied to the reference set of texts translated from L, and we compute the perplexity: the fitness of the LM to the reference set. Table 8 details the results. The lowest perplexity (reflecting the best fit) in each subcorpus is typeset in boldface, and the highest (worst fit) is italicized.

These results overwhelmingly support our hypothesis. For each language L, the perplexity of the language model that was created from L translations is lowest, followed immediately by the MIX LM. Furthermore, the perplexity of the LM created from originally-English texts is highest in all experiments (except the Dutch-to-English translation task, where the perplexity of the 2-gram LM created from texts translated from Italian is slightly higher). The perplexity of LMs constructed from texts translated from languages other than L always lies between these two extremes: It is a better fit of the reference set than original texts, but not as good as texts translated from L (or mixture translations). This gives rise to yet another hypothesis, namely, that translations from typologically related languages form a similar “translationese dialect,” whereas translations from more distant source languages form two different “dialects” in the target language (see Koppel and Ordan 2011).

A possible explanation for the different perplexity results among the LMs could be the specific content of the corpora used to compile the LMs. For example, one would expect texts translated from Dutch to exhibit higher frequencies of words such as Amsterdam or even canal. This, indeed, is reflected by the lower (usually lowest) number of OOV items in language models compiled from texts translated from the source language.

As a specific example, the top five words that occur in the T-FR corpus and the evaluation set, but are absent from the O-EN corpus, are: biarritz, meat-and-bone, armenian, ievoli, and ivorian. The top five words that occur in the O-EN corpus, but are absent from the T-FR corpus, are: duhamel, paciotti, ivoirian, coke, and spds. Of those, biarritz seems to be French-specific, but the other items seem more arbitrary.

To rule out the possibility that the perplexity results are due to specific content phenomena, and to further emphasize that the corpora are indeed structurally different, we conduct more experiments, in which we gradually abstract away from the domain- and content-specific features of the texts and emphasize their syntactic structure. We focus on French-to-English, but the results are robust and consistent (we repeated the same experiments for all language pairs, with very similar outcomes).

First, we remove all punctuation to eliminate possible bias due to differences in punctuation conventions.³ Then, we use the Stanford Named Entity Recognizer (Finkel, Grenager, and Manning 2005) to identify named entities, which we replace with a unique token (‘NE’). Next, we replace all nouns with their part-of-speech (POS) tag; we use the Stanford POS Tagger (Toutanova and Manning 2000). Finally, for full lexical abstraction, we replace all words with their POS tags, retaining only abstract syntactic structures devoid of lexical content.

At each step, we train six language models on O- and T-texts and apply them to the reference set (which is adapted to the same level of abstraction, of course). As the abstraction of the text increases, we also increase the order of the LMs: From 4-grams for text without punctuation and NE abstraction, to 5-grams for noun abstraction, to 8-grams for full POS abstraction. In all cases we fix the LM vocabulary to only contain tokens that appear more than once in the “abstracted” reference set. The results, depicted in Table 9, consistently show that the T-based LM is a better fit to the reference set, albeit to a lesser extent. The rightmost column specifies the improvement, in terms of perplexity, of each language model, compared with the worst-performing model. Although we do not show the details here, the same pattern is persistent in all the other Europarl languages we experiment with.

To further test the robustness of these phenomena, we repeat these experiments with the Hebrew-to-English corpus and reference set, reflecting a different language family, a smaller corpus, and a different domain. We train two 4-gram language models on the O-EN and T-HE corpora. We then apply the two LMs to the reference set and compute the perplexity. The results are presented in Table 10. Again, the T-based LM is a better fit to the translated text than the O-based LM: Its perplexity is lower by 12.8%. We also repeat the abstraction experiments on the Hebrew scenario. The results, depicted in Table 11, consistently show that the T-based LM is a better fit to the reference set.

Clearly, then, translated LMs better fit the references than original ones, and the differences can be traced back not just to (trivial) specific lexical choice, but also to syntactic structure, as evidenced by the POS abstraction experiments.

We further test our findings on other target languages, specifically English–German and English–French. We train several 4-gram language models on the corpora specified in Table 2. We then compute the perplexity of the German-translated-from-English and French-translated-from-English reference sets (see Section 3.4) with respect to these language models. Table 12 depicts the results; they are in complete agreement with our hypothesis.

Can these phenomena be attributed to the relatively small size of the corpora we use? Will the perplexity of O texts converge to that of T texts when more data become available, or will the differences persist? To address these questions, we use the (much larger) Hansard corpus and the (even larger) Gigaword corpus. We train 4-gram language models for each Hansard and Gigaword subcorpus described in Section 3.2. We apply the LMs to the Hansard reference set, but also to the Europarl reference set, to examine the effect on out-of-domain (but similar genre) texts. In both cases we report perplexity (Table 13).

The results are fully consistent with our previous findings: In the case of the Hansard reference set, a language model based on original texts must be up to ten times larger to retain the low perplexity level of translated texts. For example, whereas a language model compiled from 10 million English-translated-from-French tokens yields a perplexity of 42.70 on the Hansard reference set, a LM compiled from original English texts requires 100 million words to yield a similar perplexity of 43.70 on the same reference set. The Gigaword LMs, which are trained on texts representing completely different domains and genres, produce much higher (i.e., worse) perplexity in this scenario. In the case of the Europarl reference set, a language model based on original texts must be approximately five times larger (and a Gigaword language model approximately twenty times larger) than a language model based on original texts to yield similar perplexity.

The last hypothesis we test is whether a better fitting language model yields a better machine translation system. In other words, we expect the T-based LMs to outperform the O-based LMs when used as part of machine translation systems. We construct German-to-English, English-to-German, French-to-English, French-to-German, Italian-to-English, and Dutch-to-English MT systems using the Moses phrase-based SMT toolkit (Koehn et al. 2007). The systems are trained on the parallel corpora described in Section 3.3. We use the reference sets (Section 3.4) as follows: 1,000 sentences are randomly extracted for minimum error-rate training (Och 2003), and another, disjoint set of 1,000 randomly selected sentences is used for evaluation. Each system is built and tuned with six different LMs: MIX, O-based, and four T-based models (Section 3.2). We use Bleu (Papineni et al. 2002) to evaluate translation quality. The results are listed in Tables 14 and 15.

The results are consistent and fully confirm our hypothesis. Across all language pairs, MT systems using LMs compiled from translated-from-source texts consistently outperform all other systems. Systems that use LMs compiled from texts originally written in the target language always perform worst or second worst. We test the statistical significance of the differences between the results using the bootstrap resampling method (Koehn 2004). In all experiments, the best system (translated-from-source LM) is significantly better than the system that uses the O-based LM (p < 0.01).

We now repeat the experiment with Hebrew to English translation. We construct a Hebrew-to-English MT system with Moses, using a factored translation model (Koehn and Hoang 2007). Every token in the training corpus is represented as two factors: surface form and lemma. The Hebrew input is fully segmented (Itai and Wintner 2008). The system is built and tuned with O- and T-based LMs. The O-based LM yields a Bleu score of 11.94, whereas using the T-based LM results in somewhat higher Bleu score, 12.07, but the difference is not statistically significant (p = 0.18). Presumably, the low quality of both systems prevents the better LM from making a significant difference.

Again, the LMs used in the MT experiments reported here are relatively small. To assess whether the benefits of using translated LMs carry over to scenarios where larger original corpora exist, we build yet another set of French-to-English MT systems. We use the Hansard SMT translation model and Hansard LMs to train nine MT systems, three with varying sizes of translated texts and six with varying sizes of original texts. We train additional MT systems with several subsets of the Gigaword LM. We tune and evaluate on the Hansard reference set. In another set of experiments we use the Europarl French-to-English scenario (using Europarl corpora for the translation model as well as for tuning and evaluation), but we use the Hansard and Gigaword LMs to see whether our findings are consistent also when LMs are trained on out-of-domain material.

Table 16 again demonstrates that language models compiled from original texts must be up to ten times larger in order to yield translation quality similar to that of LMs compiled from translated texts.⁴ In other words, much smaller translated LMs perform better than much larger original ones, and this holds for various LM sizes, both in-domain and out-of-domain. For example, on the Hansard corpus, a 10-million-token T-FR language model yields a Bleu score of 34.67, whereas an O-EN language model of 100 million tokens is required in order to yield a similar Bleu score of 34.44. The systems that use the Gigaword LMs perform much worse in-domain, even with a language model compiled from 1000M tokens. Out-of-domain, the Gigaword systems are better than O-EN, but they require approximately five times more data to match the performance of T-FR systems.

The previous section established the fact that language models compiled from translated texts are better for MT than ones compiled from original texts, even when the original LMs are much larger. In many real-world scenarios, however, one has access to texts of both types. Our results do not imply that original texts are useless, and that only translated ones should be used. In this section we explore various ways to combine original and translated texts, thereby yielding even better language models.

For these experiments we use 10 million English-translated-from-French tokens from the Hansard corpus (T-FR) and another 100 million original-English tokens from the same source (O-EN). We combine them in five different ways: straightforward concatenation of the corpora; a concatenation of the original-English corpus with the translated corpus, upweighted by a factor of 10 and then of 20; log-linear modeling; and an interpolated language model. In each experiment we report both the fitness of the LM to the reference set, in terms of perplexity, and the quality of machine translation that uses this LM, in terms of Bleu.⁵ We execute each experiment twice, once (in-domain) with the Hansard reference set and once (out-of-domain) where the translation model, tuning corpus, and reference set all come from the Europarl FR-EN subcorpus, as above. The results are listed in Table 17; we now provide a detailed explanation of these experiments.

Concatenation of O and T texts. We train three language models by concatenating the T-FR and O-EN corpora. First, we simply concatenate the corpora obtaining 110 million tokens. Second, we upweight the T-FR corpus by a factor of 10 before the concatenation; and finally, we upweight the T-FR corpus by a factor of 20 before the concatenation. In the “in-domain” scenario, the LM trained on a simple concatenation of the corpora reduces the perplexity by more than 10%. The best translation quality is obtained when the T-FR corpus is upweighted by a factor of 10. It improves by 0.42 Bleu points compared to the MT system that uses T-FR (p = 0.074), and, more significantly, by 0.65 Bleu points compared to O-EN (p < 0.05). In the “out-of-domain” scenario, there is a small reduction in perplexity (about 5%) with a language model that is trained on a simple concatenation of the corpora. There is also a very small improvement in the translation quality (0.07 Bleu points compared to the T-FR system and 0.25 Bleu points compared to O-EN).

We train two language models, based on T-FR and O-EN. Then, we combine these models by including them as different feature functions. The feature weight of each LM is set by minimum error-rate tuning, optimizing the translation quality; this is the same technique that (Koehn and Schroeder 2007) employ for domain adaptation. In-domain, this combination is better by 0.82 Bleu points compared with an MT system that uses O-EN (p < 0.001), 0.59 Bleu points compared with the one that uses T-FR (p < 0.05). Out of domain, this combination is again not significantly better than using T-FR only (improvement of 0.08 Bleu points, p = 0.255).

In summary, LMs compiled from source-translated-to-target texts are almost as good as much larger LMs that also include large corpora of texts originally written in the target language. Clearly, ignoring the status (original or translated) of monolingual texts and creating a single language model from all of them (the concatenation scenario) is not much better than using only translated texts. In order to benefit from (often much larger) original texts, one must consider more creative ways of combining the two subcorpora. Of the methods we explored here, interpolated LMs provide the greatest advantage. More research is needed in order to find an optimal combination.

We use language models computed from different types of corpora to investigate whether their fitness to a reference set of translated sentences can differentiate between them (and, hence, between the corpora on which they are based). Our main findings are that LMs compiled from manually translated corpora are much better predictors of translated texts than LMs compiled from original-language corpora of the same size. The results are robust, and are sustainable even when the corpora and the reference sentences are abstracted in ways that retain their syntactic structure but ignore specific word meanings. Furthermore, we show that translated LMs are better predictors of translated sentences even when the LMs are compiled from texts translated from languages other than the source language. LMs based on texts translated from the source language still outperform LMs translated from other languages, however.

We also show that MT systems based on translated-from-source-language LMs outperform MT systems based on originals LMs or LMs translated from other languages. Again, these results are robust and the improvements are statistically significant. This effect seems to be amplified as translation quality improves. Furthermore, our results show that original LMs require five to ten times more data to exhibit the same fitness to the reference set and the same translation quality as translated LMs.

More generally, this study confirms that insights drawn from the field of theoretical Translation Studies, namely, the dual claim according to which translations as such differ from originals, and translations from different source languages differ from each other, can be verified experimentally and contribute to the performance of machine translation.

One question, however, requires further investigation: Do MT systems based on translated-from-source-language LMs produce better translations, or do they merely generate sentences that are directly adapted to the reference set, thereby only improving a specific evaluation metric, such as Bleu? We address this issue in three ways, showing that the former is indeed the case. First, we use two automated evaluation metrics other than Bleu, and show that the T-based LMs yield better MT systems even with different metrics. Second, we perform a manual evaluation of a portion of the evaluation set. The results show that human evaluators prefer translations produced by an MT system that uses a T-based LM over translations produced by a system built with an O-based LM. Finally, we provide a detailed analysis of the differences between O- and T-based LMs, explaining these differences in terms of insights from Translation Studies.

First, we use two alternative automatic evaluation metrics, METEOR⁶ (Denkowski and Lavie 2011) and TER (Snover et al. 2006), to assess the quality of the MT systems described in Section 4.2. We focus on four translation tasks: From German, French, Italian, and Dutch to English.⁷ For each task we report the performance of two MT systems: One that uses a language model compiled from original-English texts, and one that uses a language model trained on texts translated from the source language. The results, which are reported in Table 18, fully support our previous findings (recall that lower TER is better): MT systems that use T-based LMs significantly outperform systems that use O-based LMs.

To further establish the qualitative difference between translations produced with an English-original language model and translations produced with a LM created from French-translated-to-English texts, we conducted a human evaluation campaign, using Amazon’s Mechanical Turk as an inexpensive, reliable, and accessible pool of annotators (Callison-Burch and Dredze 2010). We created a small evaluation corpus of 100 sentences, selected randomly among all (Europarl) reference sentences whose length is between 15 and 25 words. Each instance of the evaluation task includes two English sentences, obtained from the two MT systems that use the O-EN and the T-FR language models, respectively. Annotators are presented with these two translations, and are requested to determine which one is better. The definition given to annotators is: “A better translation is more fluent, reflecting better use of English.” Observe that because the only variable that distinguishes between the two MT systems is the different language model, we only have to evaluate the fluency of the target sentence, not its faithfulness to the source. Consequently, we do not present the source or the reference translation to the annotators. All annotators were located in the United States (and, therefore, are presumably English speakers).

As a control set, we added a set of 10 sentences produced with the O-based LM, which were paired with their (manually created) reference translations, and 10 sentences produced with the T-based LM, again paired with their references. Each of the 120 evaluation instances was assigned to 10 different Mechanical Turk annotators. We report two evaluation metrics: score and majority. The score of a given sentence pair is i/j, where i is the number of annotators who preferred e₁ over e₂, and j = 10 − i is the number of annotators preferring e₂. For such a sentence pair, the majority is e₁ if i > j, e₂ if i < j, and undefined otherwise.

The average score of the 10 sentences in the O-vs.-reference control set is 22/78, and the majority is the reference translation in all but one of the instances. As for the T-vs.-reference control set, the average score is 18/82, and the majority is the reference in all of the instances. This indicates that the annotators are reliable, and also that it is unrealistic to expect a clear-cut distinction even between human translations and machine-generated output.

As for the actual evaluation set, the average score of O-EN vs. T-FR is 38/62, and the majority is T-FR in 75% of the cases, O-EN in only 25% of the sentence pairs. We take these results as a very strong indication that English sentences generated by an MT system whose language model is compiled from translated texts are perceived by humans as more fluent than ones generated by a system built with an O-based language model. Not only is the improvement reflected in significantly higher Bleu (and METEOR, TER) scores, but it is undoubtedly also perceived as such by human annotators.

In order to look into the differences between T and O qualitatively, rather than quantitatively, we turn now to study several concrete examples. To do so, we extracted approximately 200 sentences from the French–English Europarl evaluation set; we chose all sentences of length between 15 and 25. In addition, we extracted the 100 most frequent n-grams, for 1 ≤ n ≤ 5, from both English-original and English-translated-from-French Europarl corpora. As both corpora include approximately the same number of tokens, we report counts in the following rather than frequencies.

The differences between O and T texts are consistent with well-established observations of translation scholars. Consider the explicitation hypothesis (Blum-Kulka 1986), which Séguinot (1998, page 108) spells out thus:

• 1. 

  “something which was implied or understood through presupposition in the source text is overtly expressed in the translation”

• 2. 

  “something is expressed in the translation which was not in the original”

• 3. 

  “an element in the source text is given greater importance in the translation through focus, emphasis, or lexical choice”

An immediate example of (1) is the case of acronyms: these tend to be spelled out in translated texts. Indeed, the acronym EU is ranked 77 among the O-EN bigrams, whereas in T-FR it does not appear in the top 100. On the other hand, the explicit trigram The European Union occurs more frequently in T than in O.

Similarly, an instance of (2) is the cohesive marker because in the following example, which appears in T but neither appears in O nor can it be traced back to the original source sentence:

Other cohesive markers discussed by Blum-Kulka (1986) are over-represented in T compared with O. These include: therefore (3,187 occurrences in T, 1,983 in O); for example (863 occurrences in T, 701 in O); in particular (1336 vs. 1068); first of all (601 vs. 266); in fact (1014 vs. 441); in other words (553 vs. 87); with regard to (1137 vs. 310); in order to (2,016 vs. 603); in this respect (363 vs. 94); on the one hand (288 vs. 72); on the other hand (428 vs. 76); and with a view to (213 vs. 51). A similar list of markers have been shown to be excellent discriminating features between original and translated texts (from several European languages, including French) in an independent study (Koppel and Ordan 2011).

Another phenomenon we notice is that the T-based language model does a much better job translating verbs than the O-based language model. In two very large corpora of French and English (Ferraresi et al. 2008), verbs are much more frequent in French than in English (0.124 vs. 0.091). Human translations from French to English, therefore, provide many more examples of verbs from which to model. Indeed, we encounter several examples in which the O-based translation system fails to use a verb at all, or to use one correctly, compared with the T-based system:

Last, there are several cases of interference, which Toury (1995, page 275) defines as follows: “Phenomena pertaining to the make-up of the source text tend to be transferred to the target text.” In the following example, do not say nothing more is a literal translation of the French construction On ne dit rien non plus. The T-based translation is much more fluent:

Incidentally, there are also some cultural differences between O and T that we deem less important, because they are not part of the “translationese dialect” but rather indicate differences pertaining to the culture from which the speaker arrives. Most notable is the form ladies and gentlemen, which is the tenth most frequent trigram in T, but does not even rank among the top 100 in O. This is already noted by van Halteren (2008), according to whom this form is significantly more frequent in translations from five European languages as opposed to original English.

In terms of (shallow) syntactic structure, we observe that part-of-speech n-grams are distributed somewhat differently in O and in T (we use the POS-tagged Europarl corpus of Section 4.1.2 for the following analysis). For example, proper nouns are more frequent in O (ranking 7 among all POS 1-grams) than in T (rank 9). This has influence on longer n-grams: For example, the 3-gram PRP MD VB is 20% more frequent in O than in T. The sequence <S> PRP VBP is almost twice as frequent in O. The 4-gram IN DT NN </S> is 25% more frequent in O. In contrast, the 4-gram IN DT NNS IN is 15% more frequent in T than in O. A full analysis of such patterns is beyond the scope of this article.

Summing up, T-based language models are more fluent and therefore yield better translation results for the following reasons: They are more cohesive, less influenced by structural differences between the languages, such as the under-representation of verbs in original English texts, and less prone to interference (i.e., they can break away from the original towards a more coherent model of the target language).

This work is among the first to use insights from Translation Studies in order to improve machine translation, and to use computational linguistic methodologies to corroborate Translation Studies hypotheses. We believe that there are still vast opportunities for fertile cross-disciplinary research in these directions. First, we only address the language model in the present work. Kurokawa, Goutte, and Isabelle (2009) investigate the relations between the direction of translation and the quality of the translation model used by SMT systems. There are various ways in which the two approaches can be extended and combined, and we are actively pursuing such research directions now (Lembersky, Ordan, and Wintner 2012).

This work also bears on language typology: We conjecture that LMs compiled from texts translated not from the original language, but from a closely related one, can be better than LMs compiled from texts translated from a more distant language. Some of our results support this hypothesis, but more research is needed in order to establish it.

The fact that translations seem to make do with fewer words (cf. also Laviosa 2008) call into question certain norms in comparing corpora in the field of machine translation. Translated and original texts can be expected to either have the same number of sentences or the same number of tokens, but not both. Similarly, they may have the same number of tokens or the same number of types, but not both.

Another interesting question that arises from this study is whether the perplexity of a language model on a reference set is a good predictor of a translation quality measure, such as Bleu. Although our results show a certain correlation between the perplexity and Bleu, we acknowledge the fact that these results need further corroboration. Chen, Beeferman, and Rosenfeld (1998) examine the ability of perplexity to estimate the performance of speech recognition. They find that perplexity often does not correlate well with word-error rates. As it is extremely important to have a reliable measure capable of estimating the effect of language model improvements on translation quality without requiring expensive decoding resources, we believe that finding correspondences between perplexity and the quality of MT is a valuable topic for future research.

We are grateful to Cyril Goutte, George Foster, and Pierre Isabelle for providing us with an annotated version of the Hansard corpus. Alon Lavie has been instrumental in stimulating some of the ideas reported in this article, as well as in his long-term support and advice. We benefitted greatly from several constructive suggestions by the three anonymous Computational Linguistics referees. This research was supported by the Israel Science Foundation (grant no. 137/06) and by a grant from the Israeli Ministry of Science and Technology.