Skip to Main Content
Table 4: 

HANS heuristics and RoBERTa-base and SIFT’s accuracy. Examples are due to McCoy et al. (2019). “E”: entailment. “N”: non-entailment. Bold font indicates better result in each category.

HeuristicPremiseHypothesisLabelRoBERTaSIFT
Lexical The banker near the judge saw the actor. The banker saw the actor. 98.3 98.9 
Overlap The judge by the actor stopped the banker. The banker stopped the actor. 68.1 71.0 
 
Subsequence The artist and the student called the judge. The student called the judge. 99.7 99.8 
The judges heard the actors resigned. The judges heard the actors. 25.8 29.5 
 
Constituent Before the actor slept, the senator ran. The actor slept. 99.3 98.8 
If the actor slept, the judge saw the artist. The actor slept. 37.9 37.6 
HeuristicPremiseHypothesisLabelRoBERTaSIFT
Lexical The banker near the judge saw the actor. The banker saw the actor. 98.3 98.9 
Overlap The judge by the actor stopped the banker. The banker stopped the actor. 68.1 71.0 
 
Subsequence The artist and the student called the judge. The student called the judge. 99.7 99.8 
The judges heard the actors resigned. The judges heard the actors. 25.8 29.5 
 
Constituent Before the actor slept, the senator ran. The actor slept. 99.3 98.8 
If the actor slept, the judge saw the artist. The actor slept. 37.9 37.6 
Close Modal

or Create an Account

Close Modal
Close Modal