MIT Press

Table 4:

Results for acceptability classification on the CoLA test set. The first group is the CBOW baseline. The second group is the LSTM and n-gram LMs with Lau et al.’s metrics. The third group is pooling classifiers trained end-to-end on the real/fake objective. The fourth group is pooling classifiers with training on CoLA, mostly with encoders transferred from real/fake classifiers. The fifth group is the small human evaluations (Section 3). CoLA-Thresh. is threshold tuning on CoLA, and WLPM is Lau et al.’s Word LogProb Min-1 metric.

Model	Embeddings	Encoder Training	Classifier Training	In-domain		Out-of-domain
Model	Embeddings	Encoder Training	Classifier Training	Acc.	MCC	Acc.	MCC
CBOW	BNC	–	CoLA	0.502	0.063	0.482	0.096

LSTM LM WLPM	BNC	–	CoLA Thresh.	0.652	0.253	0.711	0.238
4-gram LM WLPM	–	–	CoLA Thresh.	0.474	0.000	0.645	0.042
3-gram LM WLPM	–	–	CoLA Thresh.	0.428	0.142	0.681	0.141
2-gram LM WLPM	–	–	CoLA Thresh.	0.452	0.094	0.707	0.180

Pooling Classifier	BNC	Real/Fake	Real/Fake	0.728	0.196	0.707	0.180
Pooling Classifier	GloVe	Real/Fake	Real/Fake	0.766	0.302	0.660	0.063
Pooling Classifier	ELMo-Style	Real/Fake	Real/Fake	0.758	0.265	0.702	0.177

Pooling Classifier	ELMo-Style	CoLA	CoLA	0.726	0.278	0.651	0.155
Pooling Classifier	BNC	Real/Fake	CoLA	0.723	0.261	0.679	0.186
Pooling Classifier	GloVe	Real/Fake	CoLA	0.706	0.300	0.608	0.135
Pooling Classifier	ELMo-Style	Real/Fake	CoLA	0.772	0.341	0.732	0.281

Human Average	–	–	–	0.850	0.644	0.872	0.738
Human Aggregate	–	–	–	0.870	0.695	0.910	0.815

Model	Embeddings	Encoder Training	Classifier Training	In-domain		Out-of-domain
Model	Embeddings	Encoder Training	Classifier Training	Acc.	MCC	Acc.	MCC
CBOW	BNC	–	CoLA	0.502	0.063	0.482	0.096

LSTM LM WLPM	BNC	–	CoLA Thresh.	0.652	0.253	0.711	0.238
4-gram LM WLPM	–	–	CoLA Thresh.	0.474	0.000	0.645	0.042
3-gram LM WLPM	–	–	CoLA Thresh.	0.428	0.142	0.681	0.141
2-gram LM WLPM	–	–	CoLA Thresh.	0.452	0.094	0.707	0.180

Pooling Classifier	BNC	Real/Fake	Real/Fake	0.728	0.196	0.707	0.180
Pooling Classifier	GloVe	Real/Fake	Real/Fake	0.766	0.302	0.660	0.063
Pooling Classifier	ELMo-Style	Real/Fake	Real/Fake	0.758	0.265	0.702	0.177

Pooling Classifier	ELMo-Style	CoLA	CoLA	0.726	0.278	0.651	0.155
Pooling Classifier	BNC	Real/Fake	CoLA	0.723	0.261	0.679	0.186
Pooling Classifier	GloVe	Real/Fake	CoLA	0.706	0.300	0.608	0.135
Pooling Classifier	ELMo-Style	Real/Fake	CoLA	0.772	0.341	0.732	0.281

Human Average	–	–	–	0.850	0.644	0.872	0.738
Human Aggregate	–	–	–	0.870	0.695	0.910	0.815

View Large

This Feature Is Available To Subscribers Only