Top-10 accuracy on test set. Performance increases for all languages moving from the baseline (MRR Baseline) to discriminative training (Supervised Model). The average accuracy across languages using the MRR baseline is 15.8% and using our supervised approach is 34.2%.
| Language . | MRR Baseline . | Supervised Model . | Absolute Improvement . | % Relative Improvement . |
|---|---|---|---|---|
| Vietnamese | 2.5 | 7.9 | 5.4 | 216.0 |
| Uzbek | 4.3 | 10.8 | 6.5 | 151.2 |
| Somali | 9.1 | 18.1 | 9.0 | 98.9 |
| Turkish | 9.0 | 22.5 | 13.5 | 150.0 |
| Hungarian | 8.1 | 22.6 | 14.5 | 179.0 |
| Nepali | 11.0 | 22.8 | 11.8 | 107.3 |
| Azeri | 10.7 | 25.6 | 14.9 | 139.3 |
| Cebuano | 12.3 | 28.3 | 16.0 | 130.1 |
| Indonesian | 17.4 | 32.0 | 14.6 | 83.9 |
| Swedish | 15.4 | 32.6 | 17.2 | 111.7 |
| Slovak | 13.6 | 36.6 | 23.0 | 169.1 |
| Bengali | 19.6 | 37.4 | 17.8 | 90.8 |
| Ukrainian | 13.6 | 37.7 | 24.1 | 177.2 |
| Tamil | 17.1 | 37.9 | 20.8 | 121.6 |
| Latvian | 16.6 | 38.5 | 21.9 | 131.9 |
| Albanian | 19.4 | 39.6 | 20.2 | 104.1 |
| Telugu | 25.7 | 41.0 | 15.3 | 59.5 |
| Bosnian | 19.0 | 43.1 | 24.1 | 126.8 |
| Hindi | 25.9 | 43.4 | 17.5 | 67.6 |
| Welsh | 14.5 | 44.4 | 29.9 | 206.2 |
| Gujarati | 33.3 | 45.3 | 12.0 | 36.0 |
| Serbian | 18.8 | 47.2 | 28.4 | 151.1 |
| Romanian | 17.3 | 47.6 | 30.3 | 175.1 |
| Bulgarian | 26.0 | 56.9 | 30.9 | 118.8 |
| Average | 15.8 | 34.2 | 18.3 | 129.7 |
| Language . | MRR Baseline . | Supervised Model . | Absolute Improvement . | % Relative Improvement . |
|---|---|---|---|---|
| Vietnamese | 2.5 | 7.9 | 5.4 | 216.0 |
| Uzbek | 4.3 | 10.8 | 6.5 | 151.2 |
| Somali | 9.1 | 18.1 | 9.0 | 98.9 |
| Turkish | 9.0 | 22.5 | 13.5 | 150.0 |
| Hungarian | 8.1 | 22.6 | 14.5 | 179.0 |
| Nepali | 11.0 | 22.8 | 11.8 | 107.3 |
| Azeri | 10.7 | 25.6 | 14.9 | 139.3 |
| Cebuano | 12.3 | 28.3 | 16.0 | 130.1 |
| Indonesian | 17.4 | 32.0 | 14.6 | 83.9 |
| Swedish | 15.4 | 32.6 | 17.2 | 111.7 |
| Slovak | 13.6 | 36.6 | 23.0 | 169.1 |
| Bengali | 19.6 | 37.4 | 17.8 | 90.8 |
| Ukrainian | 13.6 | 37.7 | 24.1 | 177.2 |
| Tamil | 17.1 | 37.9 | 20.8 | 121.6 |
| Latvian | 16.6 | 38.5 | 21.9 | 131.9 |
| Albanian | 19.4 | 39.6 | 20.2 | 104.1 |
| Telugu | 25.7 | 41.0 | 15.3 | 59.5 |
| Bosnian | 19.0 | 43.1 | 24.1 | 126.8 |
| Hindi | 25.9 | 43.4 | 17.5 | 67.6 |
| Welsh | 14.5 | 44.4 | 29.9 | 206.2 |
| Gujarati | 33.3 | 45.3 | 12.0 | 36.0 |
| Serbian | 18.8 | 47.2 | 28.4 | 151.1 |
| Romanian | 17.3 | 47.6 | 30.3 | 175.1 |
| Bulgarian | 26.0 | 56.9 | 30.9 | 118.8 |
| Average | 15.8 | 34.2 | 18.3 | 129.7 |