Skip to Main Content
Table 7

Evaluation of selected state-of-the-art representation learning models on the full SimVerb-3500 data set (SV-3500), the subset of pairs shared by SimVerb and our data set, using both the original SimVerb scores (SVSpA_SVs) and scores obtained via our arena-based method (SVSpA_SpAs), as well as our full similarity data set (SpA-Verb) and the thresholded subset (SpA-Verb-THR) of the whole data set (10,371 pairs from the classes with IAA ≥ 0.3). All scores are Spearman’s ρ correlations. Numbers in brackets refer to vector dimensionality.

Model (Dimensionality)SV-3500SV ∩ SpA_SVsSV ∩ SpA_SpAsSpA-VerbSpA-Verb-THR
SGNS-BOW2 (300) 0.275 0.197 0.136 0.179 0.158 
  
CBOW-CC (300) 0.365 0.264 0.242 0.271 0.305 
ELMo-Static (300) 0.414 0.327 0.310 0.230 0.227 
  
Non-distributional (172, 418) 0.606 0.543 0.479 0.295 0.310 
SGNS + Attract-Repel (300) 0.766 0.730 0.567 0.385 0.394 
  
BERT-BASE (768) (iso0.338 0.224 0.207 0.235 0.240 
context-10 0.436 0.326 0.228 0.262 0.266 
context-100 0.438 0.326 0.231 0.265 0.271 
context-500 0.439 0.327 0.231 0.265 0.270 
  
BERT-LARGE (1,024) (iso0.319 0.240 0.188 0.224 0.215 
context-10 0.403 0.305 0.226 0.255 0.269 
context-100 0.402 0.304 0.225 0.256 0.270 
context-500 0.403 0.304 0.225 0.256 0.270 
  
BERT-LARGE-WWM (1,024) (iso0.396 0.307 0.257 0.237 0.246 
Model (Dimensionality)SV-3500SV ∩ SpA_SVsSV ∩ SpA_SpAsSpA-VerbSpA-Verb-THR
SGNS-BOW2 (300) 0.275 0.197 0.136 0.179 0.158 
  
CBOW-CC (300) 0.365 0.264 0.242 0.271 0.305 
ELMo-Static (300) 0.414 0.327 0.310 0.230 0.227 
  
Non-distributional (172, 418) 0.606 0.543 0.479 0.295 0.310 
SGNS + Attract-Repel (300) 0.766 0.730 0.567 0.385 0.394 
  
BERT-BASE (768) (iso0.338 0.224 0.207 0.235 0.240 
context-10 0.436 0.326 0.228 0.262 0.266 
context-100 0.438 0.326 0.231 0.265 0.271 
context-500 0.439 0.327 0.231 0.265 0.270 
  
BERT-LARGE (1,024) (iso0.319 0.240 0.188 0.224 0.215 
context-10 0.403 0.305 0.226 0.255 0.269 
context-100 0.402 0.304 0.225 0.256 0.270 
context-500 0.403 0.304 0.225 0.256 0.270 
  
BERT-LARGE-WWM (1,024) (iso0.396 0.307 0.257 0.237 0.246 
Close Modal

or Create an Account

Close Modal
Close Modal