Skip to Main Content
Table 3: 

Performance of models on the MAGPIE, SemEval5B, and VNC Dataset as evaluated by Classification F1 score (F1;%) and Sequence Accuracy (SA;%); best performances are boldfaced; performances marked with asterisks are comparable in their differences are not statistically significant at p = 0.05 using bootstrapped samples that are estimated 105 times.

Data SplitModelMagpieSemEval5BVNC
F1SAF1SAF1SA
Random Gazetteer 86.67 76.47 67.29 50.70 82.68 70.47 
BERT 87.16 37.10 92.51 76.47 93.09 50.00 
Seq2Seq 92.70 83.21 94.41 *94.12 95.21 86.61 
BERT-BiLSTM-CRF 94.22 *87.71 93.29 92.44 95.45 85.03 
RNN-MHCA 95.51 *86.82 *94.94 93.56 *96.15 91.33 
IlliniMET 86.54 37.97 92.59 78.15 93.55 59.45 
DISC 95.02 *87.47 *95.80 *95.23 *96.97 93.31 
 
Type-aware Gazetteer 82.73 0.00 73.94 0.00 83.61 0.00 
BERT 86.27 39.70 73.37 35.19 86.85 50.86 
Seq2Seq 83.81 63.42 50.35 44.28 88.80 73.56 
BERT-BiLSTM-CRF 80.47 61.78 57.82 44.57 83.30 65.52 
RNN-MHCA 86.34 61.42 56.25 42.23 *88.74 79.02 
IlliniMET 83.58 39.68 69.49 41.94 87.97 54.60 
DISC 87.78 70.47 58.82 55.71 *89.02 80.46 
Data SplitModelMagpieSemEval5BVNC
F1SAF1SAF1SA
Random Gazetteer 86.67 76.47 67.29 50.70 82.68 70.47 
BERT 87.16 37.10 92.51 76.47 93.09 50.00 
Seq2Seq 92.70 83.21 94.41 *94.12 95.21 86.61 
BERT-BiLSTM-CRF 94.22 *87.71 93.29 92.44 95.45 85.03 
RNN-MHCA 95.51 *86.82 *94.94 93.56 *96.15 91.33 
IlliniMET 86.54 37.97 92.59 78.15 93.55 59.45 
DISC 95.02 *87.47 *95.80 *95.23 *96.97 93.31 
 
Type-aware Gazetteer 82.73 0.00 73.94 0.00 83.61 0.00 
BERT 86.27 39.70 73.37 35.19 86.85 50.86 
Seq2Seq 83.81 63.42 50.35 44.28 88.80 73.56 
BERT-BiLSTM-CRF 80.47 61.78 57.82 44.57 83.30 65.52 
RNN-MHCA 86.34 61.42 56.25 42.23 *88.74 79.02 
IlliniMET 83.58 39.68 69.49 41.94 87.97 54.60 
DISC 87.78 70.47 58.82 55.71 *89.02 80.46 
Close Modal

or Create an Account

Close Modal
Close Modal