MIT Press

Table 7:

Training models on SQuAD, as well as SQuAD combined with different adversarially created datasets. Results underlined indicate the best result per model. We report the mean and standard deviation (subscript) over 10 runs with different random seeds.

		Evaluation (Test) Dataset
Model	Training Dataset	$D_{SQuAD}$		$D_{BiDAF}$		$D_{BERT}$		$D_{RoBERTa}$
		EM	F₁	EM	F₁	EM	F₁	EM	F₁
BiDAF	$D_{SQuAD}$	56.7 _0.5	70.1 _0.3	11.6_1.0	21.3_1.1	8.6_0.6	17.3_0.8	8.3_0.7	16.8_0.5
	$D_{SQuAD}$ + $D_{BiDAF}$	56.3_0.6	69.7_0.4	14.4_0.9	24.4_0.9	15.6_1.1	24.7_1.1	14.3_0.5	23.3_0.7
	$D_{SQuAD}$ + $D_{BERT}$	56.2_0.6	69.4_0.6	14.4_0.7	24.2_0.8	15.7_0.6	25.1_0.6	13.9_0.8	22.7_0.8
	$D_{SQuAD}$ + $D_{RoBERTa}$	56.2_0.7	69.6_0.6	14.7 _0.9	24.8 _0.8	17.9 _0.5	26.7 _0.6	16.7 _1.1	25.0 _0.8

BERT	$D_{SQuAD}$	74.8_0.3	86.9_0.2	46.4_0.7	60.5_0.8	24.4_1.2	35.9_1.1	17.3_0.7	28.9_0.9
	$D_{SQuAD}$ + $D_{BiDAF}$	75.2_0.4	87.2 _0.2	52.4_0.9	66.5_0.9	40.9_1.3	51.2_1.5	32.9_0.9	44.1_0.8
	$D_{SQuAD}$ + $D_{BERT}$	75.1_0.3	87.1_0.3	54.1 _1.0	68.0 _0.8	43.7_1.1	54.1_1.3	34.7_0.7	45.7_0.8
	$D_{SQuAD}$ + $D_{RoBERTa}$	75.3 _0.4	87.1_0.3	53.0_1.1	67.1_0.8	44.1 _1.1	54.4 _0.9	36.6 _0.8	47.8 _0.5

RoBERTa	$D_{SQuAD}$	73.2_0.4	86.3_0.2	48.9_1.1	64.3_1.1	31.3_1.1	43.5_1.2	16.1_0.8	26.7_0.9
	$D_{SQuAD}$ + $D_{BiDAF}$	73.9 _0.4	86.7 _0.2	55.0_1.4	69.7_0.9	46.5_1.1	57.3_1.1	31.9_0.8	42.4_1.0
	$D_{SQuAD}$ + $D_{BERT}$	73.8_0.2	86.7 _0.2	55.4_1.0	70.1_0.9	48.9_1.0	59.0_1.2	32.9_1.3	43.7_1.4
	$D_{SQuAD}$ + $D_{RoBERTa}$	73.5_0.3	86.5_0.2	55.9 _0.7	70.6 _0.7	49.1 _1.2	59.5 _1.2	34.7 _1.0	45.9 _1.2

		Evaluation (Test) Dataset
Model	Training Dataset	$D_{SQuAD}$		$D_{BiDAF}$		$D_{BERT}$		$D_{RoBERTa}$
		EM	F₁	EM	F₁	EM	F₁	EM	F₁
BiDAF	$D_{SQuAD}$	56.7 _0.5	70.1 _0.3	11.6_1.0	21.3_1.1	8.6_0.6	17.3_0.8	8.3_0.7	16.8_0.5
	$D_{SQuAD}$ + $D_{BiDAF}$	56.3_0.6	69.7_0.4	14.4_0.9	24.4_0.9	15.6_1.1	24.7_1.1	14.3_0.5	23.3_0.7
	$D_{SQuAD}$ + $D_{BERT}$	56.2_0.6	69.4_0.6	14.4_0.7	24.2_0.8	15.7_0.6	25.1_0.6	13.9_0.8	22.7_0.8
	$D_{SQuAD}$ + $D_{RoBERTa}$	56.2_0.7	69.6_0.6	14.7 _0.9	24.8 _0.8	17.9 _0.5	26.7 _0.6	16.7 _1.1	25.0 _0.8

BERT	$D_{SQuAD}$	74.8_0.3	86.9_0.2	46.4_0.7	60.5_0.8	24.4_1.2	35.9_1.1	17.3_0.7	28.9_0.9
	$D_{SQuAD}$ + $D_{BiDAF}$	75.2_0.4	87.2 _0.2	52.4_0.9	66.5_0.9	40.9_1.3	51.2_1.5	32.9_0.9	44.1_0.8
	$D_{SQuAD}$ + $D_{BERT}$	75.1_0.3	87.1_0.3	54.1 _1.0	68.0 _0.8	43.7_1.1	54.1_1.3	34.7_0.7	45.7_0.8
	$D_{SQuAD}$ + $D_{RoBERTa}$	75.3 _0.4	87.1_0.3	53.0_1.1	67.1_0.8	44.1 _1.1	54.4 _0.9	36.6 _0.8	47.8 _0.5

RoBERTa	$D_{SQuAD}$	73.2_0.4	86.3_0.2	48.9_1.1	64.3_1.1	31.3_1.1	43.5_1.2	16.1_0.8	26.7_0.9
	$D_{SQuAD}$ + $D_{BiDAF}$	73.9 _0.4	86.7 _0.2	55.0_1.4	69.7_0.9	46.5_1.1	57.3_1.1	31.9_0.8	42.4_1.0
	$D_{SQuAD}$ + $D_{BERT}$	73.8_0.2	86.7 _0.2	55.4_1.0	70.1_0.9	48.9_1.0	59.0_1.2	32.9_1.3	43.7_1.4
	$D_{SQuAD}$ + $D_{RoBERTa}$	73.5_0.3	86.5_0.2	55.9 _0.7	70.6 _0.7	49.1 _1.2	59.5 _1.2	34.7 _1.0	45.9 _1.2

View Large

This Feature Is Available To Subscribers Only