Table 1: 

Categories of potential sources of disagreement, with examples. The last two columns give the number of annotations for each NLI label “Entailment” (E), “Neutral” (N), and “Contradiction” (C), in MNLI and ChaosNLI.

PremiseHypothesisMNLIChaosNLI
[E,N,C][E,N,C]
Uncertainty in Sentence Meaning 
[1] Lexical Technological advances generally come in waves that crest and eventually subside. Advances in electronics come in waves. [3,1,1] [82,17,1] 
[2] Implicature Today it is possible to buy cheap papyrus printed with gaudy Egyptian scenes in almost every souvenir shop in the country, but some of the most authentic are sold at The Pharaonic Village in Cairo where the papyrus is grown, processed, and hand-painted on site. The Pharaonic Village in Cairo is the only place where one can buy authentic papyrus. [0,2,3] [2,39,41] 
[3] Presupposition What changed? Nothing changed. [0,2,3] [4,76,20] 
[4] Probabilistic Enrichment It’s absurd but I can’t help it. Sir James nodded again. Sir James thinks it’s absurd. [3,2,0] [63,35,2] 
[5] Imperfection profit rather Our profit has not been good. [0,3,2] [3,90,7] 
 
Underspecification in Guidelines 
[6] Coreference This was built 15 years earlier by Jahangir’s wife, Nur Jahan, for her father, who served as Mughal Prime Minister. Nur Jahan’s husband Jahangir served as Mughal Prime Minister. [2,0,3] [24,45,31] 
[7] Temporal Reference However, co-requesters cannot approve additional co-requesters or restrict the timing of the release of the product after it is issued. They cannot restrict timing of the release of the product. [3,2,0] [90,8,2] 
[8] Interrogative Hypothesis “How did you get it?” A chair was overturned. “How did you get your hands on this object?” [3,2,0] [45,52,3] 
 
Annotator Behavior 
[9] Accommodating Minimally Added Content Indeed, 58 percent of Columbia/HCA’s beds lie empty, compared with 35 percent of nonprofit beds. 58% of Columbia/HCA’s beds are empty, said the report. [3,2,0] [97,3,0] 
[10] High Overlap Yet, in the mouths of the white townsfolk of Salisbury, N.C., it sounds convincing. White townsfolk in Salisbury, N.C. think it sounds convincing. [3,2,0] [68,27,5] 
PremiseHypothesisMNLIChaosNLI
[E,N,C][E,N,C]
Uncertainty in Sentence Meaning 
[1] Lexical Technological advances generally come in waves that crest and eventually subside. Advances in electronics come in waves. [3,1,1] [82,17,1] 
[2] Implicature Today it is possible to buy cheap papyrus printed with gaudy Egyptian scenes in almost every souvenir shop in the country, but some of the most authentic are sold at The Pharaonic Village in Cairo where the papyrus is grown, processed, and hand-painted on site. The Pharaonic Village in Cairo is the only place where one can buy authentic papyrus. [0,2,3] [2,39,41] 
[3] Presupposition What changed? Nothing changed. [0,2,3] [4,76,20] 
[4] Probabilistic Enrichment It’s absurd but I can’t help it. Sir James nodded again. Sir James thinks it’s absurd. [3,2,0] [63,35,2] 
[5] Imperfection profit rather Our profit has not been good. [0,3,2] [3,90,7] 
 
Underspecification in Guidelines 
[6] Coreference This was built 15 years earlier by Jahangir’s wife, Nur Jahan, for her father, who served as Mughal Prime Minister. Nur Jahan’s husband Jahangir served as Mughal Prime Minister. [2,0,3] [24,45,31] 
[7] Temporal Reference However, co-requesters cannot approve additional co-requesters or restrict the timing of the release of the product after it is issued. They cannot restrict timing of the release of the product. [3,2,0] [90,8,2] 
[8] Interrogative Hypothesis “How did you get it?” A chair was overturned. “How did you get your hands on this object?” [3,2,0] [45,52,3] 
 
Annotator Behavior 
[9] Accommodating Minimally Added Content Indeed, 58 percent of Columbia/HCA’s beds lie empty, compared with 35 percent of nonprofit beds. 58% of Columbia/HCA’s beds are empty, said the report. [3,2,0] [97,3,0] 
[10] High Overlap Yet, in the mouths of the white townsfolk of Salisbury, N.C., it sounds convincing. White townsfolk in Salisbury, N.C. think it sounds convincing. [3,2,0] [68,27,5] 
Close Modal

or Create an Account

Close Modal
Close Modal