Skip to Main Content
Table A.1

Evaluation results across different lexical representation extraction configurations on SimVerb-3500 and SpA-Verb data sets and the subset of shared pairs (cf. Table 7). L0 refers to the input embedding layer; ≤ Ln refers to embeddings computed by averaging representations over all Transformer layers up to and inclusive of the nth layer. For each layer averaging configuration we consider two configurations of special tokens (column spec): one where special tokens [CLS] and [SEP] are included (+) and one where they are excluded (−) from the subword embedding averaging step. All scores are Spearman’s ρ correlations.

Configuration LspecSV-3500SV ∩ SpA_SVsSV ∩ SpA_SpAsSpA-VerbSpA-Verb-THR
bert-base 
L0 0.219 0.142 0.096 0.167 0.195 
− 0.318 0.214 0.169 0.212 0.220 
L4 0.222 0.139 0.107 0.175 0.204 
− 0.338 0.221 0.195 0.239 0.246 
L6 0.204 0.130 0.093 0.158 0.187 
− 0.338 0.224 0.207 0.235 0.240 
L8 0.181 0.107 0.079 0.149 0.177 
− 0.315 0.202 0.198 0.219 0.224 
bert-large 
L0 0.214 0.141 0.087 0.156 0.175 
− 0.314 0.221 0.165 0.214 0.225 
L4 0.190 0.133 0.053 0.158 0.191 
− 0.331 0.248 0.191 0.229 0.222 
L6 0.184 0.127 0.038 0.145 0.180 
− 0.319 0.240 0.188 0.224 0.215 
L8 0.184 0.127 0.037 0.141 0.174 
− 0.319 0.240 0.193 0.221 0.214 
bert-large-wwm 
L0 0.211 0.128 0.090 0.167 0.191 
− 0.347 0.245 0.198 0.230 0.242 
L4 0.226 0.142 0.110 0.177 0.202 
− 0.390 0.295 0.249 0.248 0.258 
L6 0.210 0.131 0.104 0.160 0.189 
− 0.396 0.307 0.257 0.237 0.246 
L8 0.211 0.133 0.107 0.156 0.186 
− 0.395 0.312 0.258 0.224 0.234 
Configuration LspecSV-3500SV ∩ SpA_SVsSV ∩ SpA_SpAsSpA-VerbSpA-Verb-THR
bert-base 
L0 0.219 0.142 0.096 0.167 0.195 
− 0.318 0.214 0.169 0.212 0.220 
L4 0.222 0.139 0.107 0.175 0.204 
− 0.338 0.221 0.195 0.239 0.246 
L6 0.204 0.130 0.093 0.158 0.187 
− 0.338 0.224 0.207 0.235 0.240 
L8 0.181 0.107 0.079 0.149 0.177 
− 0.315 0.202 0.198 0.219 0.224 
bert-large 
L0 0.214 0.141 0.087 0.156 0.175 
− 0.314 0.221 0.165 0.214 0.225 
L4 0.190 0.133 0.053 0.158 0.191 
− 0.331 0.248 0.191 0.229 0.222 
L6 0.184 0.127 0.038 0.145 0.180 
− 0.319 0.240 0.188 0.224 0.215 
L8 0.184 0.127 0.037 0.141 0.174 
− 0.319 0.240 0.193 0.221 0.214 
bert-large-wwm 
L0 0.211 0.128 0.090 0.167 0.191 
− 0.347 0.245 0.198 0.230 0.242 
L4 0.226 0.142 0.110 0.177 0.202 
− 0.390 0.295 0.249 0.248 0.258 
L6 0.210 0.131 0.104 0.160 0.189 
− 0.396 0.307 0.257 0.237 0.246 
L8 0.211 0.133 0.107 0.156 0.186 
− 0.395 0.312 0.258 0.224 0.234 
Close Modal

or Create an Account

Close Modal
Close Modal