MIT Press

Skip to Main Content

Table 5:

The best setting (layer and encoding) for each contextualized word embedding model on the various tasks. Bold entries are the best performers on each task.

Model	VPC Classification		LVC Classification		NC Literality		NC Relations		AN Attributes		Phrase Type
	Layer	Encoding	Layer	Encoding	Layer	Encoding	Layer	Encoding	Layer	Encoding	Layer	Encoding
ELMo	All	Att	All	biLM	All	Att/None	Top	biLM	All	None	All	biLM
OpenAI GPT	All	None	Top	Att/None	Top	None	All	biLM	Top	None	All	biLM
BERT	All	Att	All	biLM	All	Att	All	None	All	None	All	biLM

Model	VPC Classification		LVC Classification		NC Literality		NC Relations		AN Attributes		Phrase Type
	Layer	Encoding	Layer	Encoding	Layer	Encoding	Layer	Encoding	Layer	Encoding	Layer	Encoding
ELMo	All	Att	All	biLM	All	Att/None	Top	biLM	All	None	All	biLM
OpenAI GPT	All	None	Top	Att/None	Top	None	All	biLM	Top	None	All	biLM
BERT	All	Att	All	biLM	All	Att	All	None	All	None	All	biLM