Machine Translation Results. For each metric, we underline the top scores among all models and boldface the top scores among NAR models based on the paired bootstrap test with p < 0.05 (Clark et al., 2011). EDITOR decodes 6–7% faster than LevT on Ro-En and En-De, and 33% faster on En-Ja, while achieving comparable or higher BLEU and RIBES.
| . | . | Distill . | Beam . | Params . | BLEU ↑ . | RIBES ↑ . | Latency (ms) ↓ . |
|---|---|---|---|---|---|---|---|
| Ro-En | AR (fairseq) | 4 | 64.5M | 32.0 | 83.8 | 357.14 | |
| AR (sockeye) | 4 | 64.5M | 32.3 | 83.6 | 369.82 | ||
| AR (sockeye) | 10 | 64.5M | 32.5 | 83.8 | 394.52 | ||
| AR (sockeye) | ✓ | 10 | 64.5M | 32.9 | 84.2 | 371.75 | |
| NAR: LevT | ✓ | – | 90.9M | 31.6 | 84.0 | 98.81 | |
| NAR: EDITOR | ✓ | – | 90.9M | 31.9 | 84.0 | 93.20 | |
| En-De | AR (fairseq) | 4 | 64.9M | 27.1 | 80.4 | 363.64 | |
| AR (sockeye) | 4 | 64.9M | 27.3 | 80.2 | 308.64 | ||
| AR (sockeye) | 10 | 64.9M | 27.4 | 80.3 | 332.73 | ||
| AR (sockeye) | ✓ | 10 | 64.9M | 27.6 | 80.5 | 363.52 | |
| NAR: LevT | ✓ | – | 91.1M | 26.9 | 81.0 | 113.12 | |
| NAR: EDITOR | ✓ | – | 91.1M | 26.9 | 80.9 | 105.37 | |
| En-Ja | AR (fairseq) | 4 | 62.4M | 44.9 | 85.7 | 292.40 | |
| AR (sockeye) | 4 | 62.4M | 43.4 | 85.1 | 286.83 | ||
| AR (sockeye) | 10 | 62.4M | 43.5 | 85.3 | 311.38 | ||
| AR (sockeye) | ✓ | 10 | 62.4M | 42.7 | 85.1 | 295.32 | |
| NAR: LevT | ✓ | – | 106.1M | 42.4 | 84.5 | 143.88 | |
| NAR: EDITOR | ✓ | – | 106.1M | 42.3 | 85.1 | 96.62 | |
| . | . | Distill . | Beam . | Params . | BLEU ↑ . | RIBES ↑ . | Latency (ms) ↓ . |
|---|---|---|---|---|---|---|---|
| Ro-En | AR (fairseq) | 4 | 64.5M | 32.0 | 83.8 | 357.14 | |
| AR (sockeye) | 4 | 64.5M | 32.3 | 83.6 | 369.82 | ||
| AR (sockeye) | 10 | 64.5M | 32.5 | 83.8 | 394.52 | ||
| AR (sockeye) | ✓ | 10 | 64.5M | 32.9 | 84.2 | 371.75 | |
| NAR: LevT | ✓ | – | 90.9M | 31.6 | 84.0 | 98.81 | |
| NAR: EDITOR | ✓ | – | 90.9M | 31.9 | 84.0 | 93.20 | |
| En-De | AR (fairseq) | 4 | 64.9M | 27.1 | 80.4 | 363.64 | |
| AR (sockeye) | 4 | 64.9M | 27.3 | 80.2 | 308.64 | ||
| AR (sockeye) | 10 | 64.9M | 27.4 | 80.3 | 332.73 | ||
| AR (sockeye) | ✓ | 10 | 64.9M | 27.6 | 80.5 | 363.52 | |
| NAR: LevT | ✓ | – | 91.1M | 26.9 | 81.0 | 113.12 | |
| NAR: EDITOR | ✓ | – | 91.1M | 26.9 | 80.9 | 105.37 | |
| En-Ja | AR (fairseq) | 4 | 62.4M | 44.9 | 85.7 | 292.40 | |
| AR (sockeye) | 4 | 62.4M | 43.4 | 85.1 | 286.83 | ||
| AR (sockeye) | 10 | 62.4M | 43.5 | 85.3 | 311.38 | ||
| AR (sockeye) | ✓ | 10 | 62.4M | 42.7 | 85.1 | 295.32 | |
| NAR: LevT | ✓ | – | 106.1M | 42.4 | 84.5 | 143.88 | |
| NAR: EDITOR | ✓ | – | 106.1M | 42.3 | 85.1 | 96.62 | |