MIT Press

Table 4:

BLEU scores for unsupervised NMT performance on WMT’14 English-French and WMT’16 English-German test sets. All methods only use unaligned Wikipedia corpora for pre-training and/or bitext mining. ‘ST’ refers to self-training.

Reference	Architecture	Pre-training	En-De	De-En	En-Fr	Fr-En
Artetxe et al. (2018)	2-layer RNN		6.89	10.16	15.13	15.56
Lample et al. (2018a)	3-layer RNN		9.75	13.33	15.05	14.31
Yang et al. (2018)	4-layer Transformer		10.86	14.62	16.97	15.58
Lample et al. (2018c)	4-layer Transformer		17.16	21.00	25.14	24.18
Song et al. (2019)	6-layer Transformer	MASS	28.3	35.2	37.5	34.9

XLM Baselines
Lample and Conneau (2019)	6-layer Transformer	XLM	–	–	33.4	33.3
Song et al. (2019)	6-layer Transformer	XLM	27.0	34.3	33.4	33.3
XLM reference implementation	6-layer Transformer	XLM	–	–	36.6	34.0
Maximum performance across baselines	6-layer Transformer	XLM	27.0	34.3	36.6	34.0

Ours
Our XLM baseline	6-layer Transformer	XLM	27.7	34.5	36.7	34.5
w/ pseudo-parallel text before ST	6-layer Transformer	XLM	30.4	36.3	39.7	35.9
w/ pseudo-parallel text after ST	6-layer Transformer	XLM	30.7	37.3	40.2	36.9

Reference	Architecture	Pre-training	En-De	De-En	En-Fr	Fr-En
Artetxe et al. (2018)	2-layer RNN		6.89	10.16	15.13	15.56
Lample et al. (2018a)	3-layer RNN		9.75	13.33	15.05	14.31
Yang et al. (2018)	4-layer Transformer		10.86	14.62	16.97	15.58
Lample et al. (2018c)	4-layer Transformer		17.16	21.00	25.14	24.18
Song et al. (2019)	6-layer Transformer	MASS	28.3	35.2	37.5	34.9

XLM Baselines
Lample and Conneau (2019)	6-layer Transformer	XLM	–	–	33.4	33.3
Song et al. (2019)	6-layer Transformer	XLM	27.0	34.3	33.4	33.3
XLM reference implementation	6-layer Transformer	XLM	–	–	36.6	34.0
Maximum performance across baselines	6-layer Transformer	XLM	27.0	34.3	36.6	34.0

Ours
Our XLM baseline	6-layer Transformer	XLM	27.7	34.5	36.7	34.5
w/ pseudo-parallel text before ST	6-layer Transformer	XLM	30.4	36.3	39.7	35.9
w/ pseudo-parallel text after ST	6-layer Transformer	XLM	30.7	37.3	40.2	36.9