MIT Press

Table 3:

Performance of models on the MAGPIE, SemEval5B, and VNC Dataset as evaluated by Classification F1 score (F1;%) and Sequence Accuracy (SA;%); best performances are boldfaced; performances marked with asterisks are comparable in their differences are not statistically significant at p = 0.05 using bootstrapped samples that are estimated 10⁵ times.

Data Split	Model	Magpie		SemEval5B		VNC
Data Split	Model	F1	SA	F1	SA	F1	SA
Random	Gazetteer	86.67	76.47	67.29	50.70	82.68	70.47
	BERT	87.16	37.10	92.51	76.47	93.09	50.00
	Seq2Seq	92.70	83.21	94.41	^*94.12	95.21	86.61
	BERT-BiLSTM-CRF	94.22	^*87.71	93.29	92.44	95.45	85.03
	RNN-MHCA	95.51	^*86.82	^*94.94	93.56	^*96.15	91.33
	IlliniMET	86.54	37.97	92.59	78.15	93.55	59.45
	DISC	95.02	^*87.47	^*95.80	^*95.23	^*96.97	93.31

Type-aware	Gazetteer	82.73	0.00	73.94	0.00	83.61	0.00
	BERT	86.27	39.70	73.37	35.19	86.85	50.86
	Seq2Seq	83.81	63.42	50.35	44.28	88.80	73.56
	BERT-BiLSTM-CRF	80.47	61.78	57.82	44.57	83.30	65.52
	RNN-MHCA	86.34	61.42	56.25	42.23	^*88.74	79.02
	IlliniMET	83.58	39.68	69.49	41.94	87.97	54.60
	DISC	87.78	70.47	58.82	55.71	^*89.02	80.46

Data Split	Model	Magpie		SemEval5B		VNC
Data Split	Model	F1	SA	F1	SA	F1	SA
Random	Gazetteer	86.67	76.47	67.29	50.70	82.68	70.47
	BERT	87.16	37.10	92.51	76.47	93.09	50.00
	Seq2Seq	92.70	83.21	94.41	^*94.12	95.21	86.61
	BERT-BiLSTM-CRF	94.22	^*87.71	93.29	92.44	95.45	85.03
	RNN-MHCA	95.51	^*86.82	^*94.94	93.56	^*96.15	91.33
	IlliniMET	86.54	37.97	92.59	78.15	93.55	59.45
	DISC	95.02	^*87.47	^*95.80	^*95.23	^*96.97	93.31

Type-aware	Gazetteer	82.73	0.00	73.94	0.00	83.61	0.00
	BERT	86.27	39.70	73.37	35.19	86.85	50.86
	Seq2Seq	83.81	63.42	50.35	44.28	88.80	73.56
	BERT-BiLSTM-CRF	80.47	61.78	57.82	44.57	83.30	65.52
	RNN-MHCA	86.34	61.42	56.25	42.23	^*88.74	79.02
	IlliniMET	83.58	39.68	69.49	41.94	87.97	54.60
	DISC	87.78	70.47	58.82	55.71	^*89.02	80.46

View Large

This Feature Is Available To Subscribers Only