Skip to Main Content
Table 1

Performance of GloVe word-level BiLSTM-CRF and BERT. All rows are for the former and only the last two rows for BERT. Local context refers to high precision constraints due to sequential CRF. Non-local context refers to the entire sentence. No document level context is included. The first two panels were trained on the Original English CoNLL 03 training data and tested on the original English CoNLL 03 test data and the WikiGold data. The last panel was trained and tested on the respective splits of MUC-6. Highest F1 in each panel is boldfaced, excluding the full systems.

SystemCoNLLWikipediaMUC-6
PRF1PRF1PRF1
Full system 
BI context + word + CRF 90.7 91.3 91.0 66.6 60.8 63.6 90.1 91.8 90.9 
  
Words only 
Lookup 84.1 56.6 67.7 66.3 28.5 39.8 81.4 54.4 65.2 
LogReg 80.2 74.3 77.2 58.8 48.9 53.4 75.1 71.7 73.4 
  
Words + local context 
GloVe fixed + CRF 67.9 63.4 65.6 53.7 37.6 44.2 74.1 68.1 70.9 
GloVe fine-tuned + CRF 80.8 77.3 79.0 63.3 45.8 53.1 82.1 77.0 79.5 
  
Non-local context-only 
FW context-only + CRF 71.3 39.4 50.8 53.3 19.3 28.4 71.9 58.9 64.7 
BW context-only + CRF 69.5 47.7 56.6 46.6 21.7 29.6 74.0 49.4 59.2 
BI context-only + CRF 70.1 52.1 59.8 51.2 21.4 30.2 66.4 56.5 61.1 
  
BERT 
Full 91.9 93.1 92.5 75.4 75.1 75.2 96.1 97.2 96.7 
Word-only 80.0 80.5 80.3 61.6 54.8 58.0 77.9 75.3 76.5 
Context-only 43.1 64.1 51.6 39.7 76.2 52.2 75.6 71.6 73.5 
SystemCoNLLWikipediaMUC-6
PRF1PRF1PRF1
Full system 
BI context + word + CRF 90.7 91.3 91.0 66.6 60.8 63.6 90.1 91.8 90.9 
  
Words only 
Lookup 84.1 56.6 67.7 66.3 28.5 39.8 81.4 54.4 65.2 
LogReg 80.2 74.3 77.2 58.8 48.9 53.4 75.1 71.7 73.4 
  
Words + local context 
GloVe fixed + CRF 67.9 63.4 65.6 53.7 37.6 44.2 74.1 68.1 70.9 
GloVe fine-tuned + CRF 80.8 77.3 79.0 63.3 45.8 53.1 82.1 77.0 79.5 
  
Non-local context-only 
FW context-only + CRF 71.3 39.4 50.8 53.3 19.3 28.4 71.9 58.9 64.7 
BW context-only + CRF 69.5 47.7 56.6 46.6 21.7 29.6 74.0 49.4 59.2 
BI context-only + CRF 70.1 52.1 59.8 51.2 21.4 30.2 66.4 56.5 61.1 
  
BERT 
Full 91.9 93.1 92.5 75.4 75.1 75.2 96.1 97.2 96.7 
Word-only 80.0 80.5 80.3 61.6 54.8 58.0 77.9 75.3 76.5 
Context-only 43.1 64.1 51.6 39.7 76.2 52.2 75.6 71.6 73.5 
Close Modal

or Create an Account

Close Modal
Close Modal