Skip to Main Content
Table 6

F1 scores obtained by representation models on the clustering task, for the optimal value of k (F1 optimal) and for k = KGold (F1 gold), evaluated against Phase 1 classes. For BERT-BASE and BERT-LARGE models, we evaluate both the embeddings computed in isolation (iso) and in context, for three values of N (10, 100, 500), corresponding to the number of contextualized representations aggregated into the final word-level embedding. Numbers in brackets refer to vector dimensionality.

Model (Dimensionality)F1 optimalF1 gold
SGNS-BOW2 (300) 0.355 0.326 
  
CBOW-CC (300) 0.426 0.383 
ELMo-Static (300) 0.394 0.387 
  
Non-distributional (172, 418) 0.391 0.360 
SGNS + Attract-Repel (300) 0.392 0.354 
BOW2-VN (300) 0.416 0.404 
BOW2-FN (300) 0.444 0.429 
  
BERT-BASE (768) (iso0.338 0.310 
context-10 0.338 0.312 
context-100 0.340 0.322 
context-500 0.332 0.309 
  
BERT-LARGE (1,024) (iso0.297 0.269 
context-10 0.339 0.325 
context-100 0.334 0.304 
context-500 0.350 0.323 
  
BERT-LARGE-WWM (1,024) (iso0.323 0.308 
Model (Dimensionality)F1 optimalF1 gold
SGNS-BOW2 (300) 0.355 0.326 
  
CBOW-CC (300) 0.426 0.383 
ELMo-Static (300) 0.394 0.387 
  
Non-distributional (172, 418) 0.391 0.360 
SGNS + Attract-Repel (300) 0.392 0.354 
BOW2-VN (300) 0.416 0.404 
BOW2-FN (300) 0.444 0.429 
  
BERT-BASE (768) (iso0.338 0.310 
context-10 0.338 0.312 
context-100 0.340 0.322 
context-500 0.332 0.309 
  
BERT-LARGE (1,024) (iso0.297 0.269 
context-10 0.339 0.325 
context-100 0.334 0.304 
context-500 0.350 0.323 
  
BERT-LARGE-WWM (1,024) (iso0.323 0.308 
Close Modal

or Create an Account

Close Modal
Close Modal