Skip to Main Content
Table 7: 

Language-wise breakdown for TyDi QA primary tasks. English is parenthesized because it is not included in the overall score calculation for TyDi QA.

LanguagemBertCanine-SCanine-CCanine-C + n-grams
SelectP 
(English) 62.2 58.6 (–3.6) 61.6 (–0.6) 64.6 (+2.4) 
Arabic 82.3 82.8 (+0.5) 82.5 (+0.2) 84.3 (+2.0) 
Bengali 58.5 61.8 (+3.3) 62.5 (+4.0) 66.0 (+7.5) 
Finnish 60.4 62.2 (+1.8) 63.6 (+3.2) 66.7 (+6.3) 
Indonesian 61.3 63.5 (+2.2) 64.2 (+2.9) 65.9 (+4.6) 
Japanese 46.2 51.7 (+5.5) 49.7 (+3.5) 51.2 (+5.0) 
Korean 60.2 60.3 (+0.1) 59.7 (–0.5) 60.6 (+0.4) 
Russian 62.2 64.6 (+2.4) 65.6 (+3.4) 68.5 (+6.3) 
Swahili 58.8 67.8 (+9.0) 67.0 (+8.2) 67.2 (+8.4) 
Telugu 81.0 82.5 (+1.5) 81.1 (+0.1) 84.6 (+3.6) 
Thai 61.1 62.8 (+1.7) 61.2 (+0.1) 65.8 (+4.7) 
Macro Avg 63.2 66.0 (+2.8) 65.7 (+2.5) 68.1 (+4.9) 
 
MinSpan 
(English) 46.0 46.3 (+0.3) 49.0 (+3.0) 51.8 (+5.8) 
Arabic 70.7 66.9 (–3.8) 65.6 (–5.1) 73.0 (+2.3) 
Bengali 47.3 46.7 (–0.6) 52.5 (+5.2) 57.1 (+9.8) 
Finnish 51.1 53.0 (+1.9) 53.8 (+2.7) 57.1 (+6.0) 
Indonesian 52.2 53.6 (+1.4) 54.4 (+2.2) 56.8 (+4.6) 
Japanese 36.1 40.3 (+4.2) 40.7 (+4.6) 42.0 (+5.9) 
Korean 36.8 35.7 (–1.1) 36.5 (–0.3) 39.9 (+3.1) 
Russian 45.6 46.7 (+1.1) 47.2 (+1.6) 51.5 (+5.9) 
Swahili 49.4 59.0 (+9.6) 57.6 (+8.2) 59.2 (+9.8) 
Telugu 75.6 75.2 (–0.4) 74.2 (–1.4) 79.7 (+4.1) 
Thai 48.4 47.9 (–0.5) 47.1 (–1.3) 54.2 (+5.8) 
Macro Avg 51.3 52.5 (+1.2) 53.0 (+1.7) 57.0 (+5.7) 
LanguagemBertCanine-SCanine-CCanine-C + n-grams
SelectP 
(English) 62.2 58.6 (–3.6) 61.6 (–0.6) 64.6 (+2.4) 
Arabic 82.3 82.8 (+0.5) 82.5 (+0.2) 84.3 (+2.0) 
Bengali 58.5 61.8 (+3.3) 62.5 (+4.0) 66.0 (+7.5) 
Finnish 60.4 62.2 (+1.8) 63.6 (+3.2) 66.7 (+6.3) 
Indonesian 61.3 63.5 (+2.2) 64.2 (+2.9) 65.9 (+4.6) 
Japanese 46.2 51.7 (+5.5) 49.7 (+3.5) 51.2 (+5.0) 
Korean 60.2 60.3 (+0.1) 59.7 (–0.5) 60.6 (+0.4) 
Russian 62.2 64.6 (+2.4) 65.6 (+3.4) 68.5 (+6.3) 
Swahili 58.8 67.8 (+9.0) 67.0 (+8.2) 67.2 (+8.4) 
Telugu 81.0 82.5 (+1.5) 81.1 (+0.1) 84.6 (+3.6) 
Thai 61.1 62.8 (+1.7) 61.2 (+0.1) 65.8 (+4.7) 
Macro Avg 63.2 66.0 (+2.8) 65.7 (+2.5) 68.1 (+4.9) 
 
MinSpan 
(English) 46.0 46.3 (+0.3) 49.0 (+3.0) 51.8 (+5.8) 
Arabic 70.7 66.9 (–3.8) 65.6 (–5.1) 73.0 (+2.3) 
Bengali 47.3 46.7 (–0.6) 52.5 (+5.2) 57.1 (+9.8) 
Finnish 51.1 53.0 (+1.9) 53.8 (+2.7) 57.1 (+6.0) 
Indonesian 52.2 53.6 (+1.4) 54.4 (+2.2) 56.8 (+4.6) 
Japanese 36.1 40.3 (+4.2) 40.7 (+4.6) 42.0 (+5.9) 
Korean 36.8 35.7 (–1.1) 36.5 (–0.3) 39.9 (+3.1) 
Russian 45.6 46.7 (+1.1) 47.2 (+1.6) 51.5 (+5.9) 
Swahili 49.4 59.0 (+9.6) 57.6 (+8.2) 59.2 (+9.8) 
Telugu 75.6 75.2 (–0.4) 74.2 (–1.4) 79.7 (+4.1) 
Thai 48.4 47.9 (–0.5) 47.1 (–1.3) 54.2 (+5.8) 
Macro Avg 51.3 52.5 (+1.2) 53.0 (+1.7) 57.0 (+5.7) 
Close Modal

or Create an Account

Close Modal
Close Modal