Skip to Main Content
Table 1: 
Examples for our reasoning probes. We use two types of experimental setups, explained in §2. A. is the correct answer.
Probe nameSetupExampleHuman
Always-Never MC-MLM A chicken [MASK] has horns.A. never B. rarely C. sometimes D. often E. always 91% 
Age Comparison MC-MLM A 21 year old person is [MASK] than me in age, If I am a 35 year old person.A. younger B. older 100% 
Objects Comparison MC-MLM The size of a airplane is [MASK] than the size of a house .A. larger B. smaller 100% 
Antonym Negation MC-MLM It was [MASK] hot, it was really cold .A. not B. really 90% 
Property Conjunction MC-QA What is usually located at hand and used for writing?A. pen B. spoon C. computer 92% 
Taxonomy Conjunction MC-MLM A ferry and a floatplane are both a type of [MASK].A. vehicle B. airplane C. boat 85% 
Encyc. Composition MC-QA When did the band where Junior Cony played first form?A. 1978 B. 1977 C. 1980 85% 
Multi-hop Composition MC-MLM When comparing a 23, a 38 and a 31 year old, the [MASK] is oldestA. second B. first C. third 100% 
Probe nameSetupExampleHuman
Always-Never MC-MLM A chicken [MASK] has horns.A. never B. rarely C. sometimes D. often E. always 91% 
Age Comparison MC-MLM A 21 year old person is [MASK] than me in age, If I am a 35 year old person.A. younger B. older 100% 
Objects Comparison MC-MLM The size of a airplane is [MASK] than the size of a house .A. larger B. smaller 100% 
Antonym Negation MC-MLM It was [MASK] hot, it was really cold .A. not B. really 90% 
Property Conjunction MC-QA What is usually located at hand and used for writing?A. pen B. spoon C. computer 92% 
Taxonomy Conjunction MC-MLM A ferry and a floatplane are both a type of [MASK].A. vehicle B. airplane C. boat 85% 
Encyc. Composition MC-QA When did the band where Junior Cony played first form?A. 1978 B. 1977 C. 1980 85% 
Multi-hop Composition MC-MLM When comparing a 23, a 38 and a 31 year old, the [MASK] is oldestA. second B. first C. third 100% 
Close Modal

or Create an Account

Close Modal
Close Modal