MIT Press

Table 4:

Monolingual evaluation: Exact match accuracies on MCWQ. MCD_mean is the mean accuracy of all three MCD splits. Random represents a random split of MCWQ. This is an upper bound on the performance shown only for comparison. As SPARQL BLEU scores are highly correlated with accuracies in this experiment, we only show the latter here.

Exact Match (%)	MCD₁				MCD₂				MCD₃				MCD_mean				Random
Exact Match (%)	En	He	Kn	Zh	En	He	Kn	Zh	En	He	Kn	Zh	En	He	Kn	Zh	En	He	Kn	Zh
LSTM+Attention	38.2	29.3	27.1	26.1	6.3	5.6	9.9	7.5	13.6	11.5	15.7	15.1	19.4	15.5	17.6	16.2	96.6	80.8	88.7	86.8
E. Transformer	53.3	35	30.7	31	16.5	8.7	11.9	10.2	18.2	13	18.1	15.5	29.3	18.9	20.2	18.9	99	90.4	93.7	92.2
mBERT	49.5	38.7	34.4	35.6	13.4	11.4	12.3	15.1	17	18	18.1	19.4	26.6	22.7	21.6	23.4	98.7	91	95.1	93.3
T5-base+RIR	57.4	–	–	–	14.6	–	–	–	12.3	–	–	–	28.1	–	–	–	98.5	–	–	–
mT5-small+RIR	77.6	57.8	55	52.8	13	12.6	8.2	21.1	24.3	17.5	31.4	34.9	38.3	29.3	31.5	36.3	98.6	90	93.8	91.8
mT5-base+RIR	55.5	59.5	49.1	30.2	27.7	16.6	16.6	23	18.2	23.4	30.5	35.6	33.8	33.2	32.1	29.6	99.1	90.6	94.2	92.2

Exact Match (%)	MCD₁				MCD₂				MCD₃				MCD_mean				Random
Exact Match (%)	En	He	Kn	Zh	En	He	Kn	Zh	En	He	Kn	Zh	En	He	Kn	Zh	En	He	Kn	Zh
LSTM+Attention	38.2	29.3	27.1	26.1	6.3	5.6	9.9	7.5	13.6	11.5	15.7	15.1	19.4	15.5	17.6	16.2	96.6	80.8	88.7	86.8
E. Transformer	53.3	35	30.7	31	16.5	8.7	11.9	10.2	18.2	13	18.1	15.5	29.3	18.9	20.2	18.9	99	90.4	93.7	92.2
mBERT	49.5	38.7	34.4	35.6	13.4	11.4	12.3	15.1	17	18	18.1	19.4	26.6	22.7	21.6	23.4	98.7	91	95.1	93.3
T5-base+RIR	57.4	–	–	–	14.6	–	–	–	12.3	–	–	–	28.1	–	–	–	98.5	–	–	–
mT5-small+RIR	77.6	57.8	55	52.8	13	12.6	8.2	21.1	24.3	17.5	31.4	34.9	38.3	29.3	31.5	36.3	98.6	90	93.8	91.8
mT5-base+RIR	55.5	59.5	49.1	30.2	27.7	16.6	16.6	23	18.2	23.4	30.5	35.6	33.8	33.2	32.1	29.6	99.1	90.6	94.2	92.2

View Large

This Feature Is Available To Subscribers Only