Skip to Main Content
Table 4: 
Examples of all 67 paradigms in BLiMP along with model performance and estimated human agreement.
PhenomenonUID5-gramLSTMTXLGPT-2HumanAcceptable ExampleUnacceptable Example1pfx2pfx
Anaphor agreement anaphor_gender_agreement 44 88 91 99 96 Katherine can’t help herselfKatherine can’t help himself  
anaphor_number_agreement 52 95 97 100 99 Many teenagers were helping themselvesMany teenagers were helping herself  
 
Argument structure animate_subject_passive 54 68 58 77 98 Amanda was respected by some waitressesAmanda was respected by some picture  
animate_subject_trans 72 79 70 80 87 Danielle visited Irene. The eye visited Irene.   
causative 51 65 54 68 82 Aaron breaks the glass. Aaron appeared the glass.   
drop_argument 68 79 67 84 90 The Lutherans couldn’t skate around. The Lutherans couldn’t disagree with.   
inchoative 89 72 81 90 95 A screen was fading. A screen was cleaning.   
intransitive 82 73 81 90 86 Some glaciers are vaporizing. Some glaciers are scaring.   
passive_1 71 65 76 89 99 Jeffrey’s sons are insulted by Tina’s supervisor. Jeffrey’s sons are smiled by Tina’s supervisor.   
passive_2 70 72 74 79 86 Most cashiers are disliked. Most cashiers are flirted.   
transitive 91 87 89 49 87 A lot of actresses’ nieces have toured that art gallery. A lot of actresses’ nieces have coped that art gallery.   
 
Binding principle_A_c_command 58 59 61 100 86 A lot of actresses that thought about Alice healed themselvesA lot of actresses that thought about Alice healed herself  
principle_A_case_1 100 100 100 96 98 Tara thinks that she sounded like Wayne. Tara thinks that herself sounded like Wayne.   
principle_A_case_2 49 87 95 73 96 Stacy imagines herself praising this actress. Stacy imagines herself praises this actress.   
principle_A_domain_1 95 98 99 99 95 Carlos said that Lori helped himCarlos said that Lori helped himself  
principle_A_domain_2 56 68 70 73 75 Mark imagines Erin might admire herselfMark imagines Erin might admire himself  
principle_A_domain_3 52 55 60 82 83 Nancy could say every guy hides himselfEvery guy could say Nancy hides himself  
principle_A_reconstruction 40 46 38 37 78 It’s herself who Karen criticized. It’s herself who criticized Karen.   
 
Control/ raising existential_there_object_raising 84 66 76 92 90 William has declared there to be no guests getting fired. William has obliged there to be no guests getting fired.   
existential_there_subject_raising 77 80 79 89 88 There was bound to be a fish escaping. There was unable to be a fish escaping.   
expletive_it_object_raising 72 63 72 58 86 Regina wanted it to be obvious that Maria thought about Anna. Regina forced it to be obvious that Maria thought about Anna.   
tough_vs_raising_1 33 34 45 72 75 Julia wasn’t fun to talk to. Julia wasn’t unlikely to talk to.   
tough_vs_raising_2 77 93 86 92 81 Rachel was apt to talk to Alicia. Rachel was exciting to talk to Alicia.   
 
Deter-miner-noun agr. determiner_noun_agreement_1 88 92 92 100 96 Craig explored that grocery storeCraig explored that grocery stores  
determiner_noun_agreement_2 86 92 81 93 95 Carl cures those horsesCarl cures that horses  
determiner_noun_agreement_irregular_1 85 82 88 94 92 Phillip was lifting this mousePhillip was lifting this mice  
determiner_noun_agreement_irregular_2 90 86 82 93 85 Those ladies walk through those oasesThose ladies walk through that oases  
determiner_noun_agreement_with_adj_1 50 86 78 90 96 Tracy praises those lucky guysTracy praises those lucky guys  
determiner_noun_agreement_with_adj_2 53 76 81 96 94 Some actors buy these gray booksSome actors buy this gray books  
determiner_noun_agreement_with_adj_irregular_1 55 83 77 88 85 This person shouldn’t criticize this upset childThis person shouldn’t criticize this upset children  
determiner_noun_agreement_with_adj_irregular_2 52 87 86 93 95 That adult has brought that purple octopusThat adult has brought those purple octopus  
 
Ellipsis ellipsis_n_bar_1 23 68 65 88 92 Brad passed one big museum and Eva passed several. Brad passed one museum and Eva passed several big.   
ellipsis_n_bar_2 50 67 89 86 78 Curtis’s boss discussed four sons and Andrew discussed five sick sons. Curtis’s boss discussed four happy sons and Andrew discussed five sick.   
 
Filler gap wh_questions_object_gap 53 79 61 84 85 Joel discovered the vase that Patricia might take. Joel discovered what Patricia might take the vase.   
wh_questions_subject_gap 82 92 83 95 98 Cheryl thought about some dog that upset Sandra. Cheryl thought about who some dog upset Sandra.   
wh_questions_subject_gap_long_distance 86 96 86 88 85 Bruce knows that person that Dawn likes that argued about a lot of guys. Bruce knows who that person that Dawn likes argued about a lot of guys.   
wh_vs_that_no_gap 83 97 86 97 97 Danielle finds out that many organizations have alarmed Chad. Danielle finds out who many organizations have alarmed Chad.   
wh_vs_that_no_gap_long_distance 81 97 91 94 92 Christina forgot that all plays that win worry Dana. Christina forgot who all plays that win worry Dana.   
wh_vs_that_with_gap 18 43 42 56 77 Nina has learned who most men sound like. Nina has learned that most men sound like.   
wh_vs_that_with_gap_long_distance 20 14 17 56 75 Martin did find out what every cashier that shouldn’t drink wore. Martin did find out that every cashier that shouldn’t drink wore.   
 
Irregular forms irregular_past_participle_adjectives 79 93 91 78 99 The forgotten newspaper article was bad. The forgot newspaper article was bad.   
irregular_past_participle_verbs 80 85 66 90 95 Edward hid the cats. Edward hidden the cats.   
 
Island effects adjunct_island 48 67 65 91 94 Who has Colleen aggravated before kissing Judy? Who has Colleen aggravated Judy before kissing?   
complex_NP__island 50 47 58 72 80 Who hadn’t some driver who would fire Jennifer’s colleague embarrassed? Who hadn’t Jennifer’s colleague embarrassed some driver who would fire?   
coordinate_structure_constraint_complex_left_branch 32 30 36 42 90 What lights could Spain sell and Andrea discover? What could Spain sell lights and Andrea discover?   
coordinate_structure_constraint_object_extraction 59 71 74 88 91 Who will Elizabeth and Gregory cure? Who will Elizabeth cure and Gregory?   
left_branch_island_echo_question 96 32 63 77 91 David would cure what snakeWhat would David cure snake  
left_branch_island_simple_question 57 36 36 82 99 Whose hat should Tonya wear? Whose should Tonya wear hat?   
sentential_subject_island 61 43 37 35 61 Who have many women’s touring Spain embarrassed. Who have many women’s touring embarrassed Spain.   
wh_island 56 47 20 77 73 What could Alan discover he has run around? What could Alan discover who has run around?   
 
NPI licensing matrix_question_npi_licensor_present 67 98 Should Monica ever grin? Monica should ever grin.   
npi_present_1 47 54 61 55 83 Even these trucks have often slowed. Even these trucks have ever slowed.   
npi_present_2 47 54 48 62 98 Many skateboards also roll. Many skateboards ever roll.   
only_npi_licensor_present 57 93 80 100 92 Only Bill would ever complain. Even Bill would ever complain.   
only_npi_scope 30 36 45 85 72 Only those doctors who Karla respects ever conceal many snakes. Those doctors who only Karla respects ever conceal many snakes.   
sentential_negation_npi_licensor_present 93 100 99 89 93 Those banks had not ever lied. Those banks had really ever lied.   
sentential_negation_npi_scope 45 23 53 95 81 Those turtles that are boring April could not ever break those couches. Those turtles that are not boring April could ever break those couches.   
 
Quantifiers existential_there_quantifiers_1 91 96 94 99 94 There aren’t many lights darkening. There aren’t all lights darkening.   
existential_there_quantifiers_2 62 16 14 24 76 Each book is there disturbing Margaret. There is each book disturbing Margaret.   
superlative_quantifiers_1 45 63 84 84 91 No man has revealed more than five forks. No man has revealed at least five forks.   
superlative_quantifiers_2 17 83 85 78 85 An actor arrived at at most six lakes. No actor arrived at at most six lakes.   
 
Subject-verb agr. distractor_agreement_relational_noun 24 76 77 83 81 A sketch of lights doesn’t appear. A sketch of lights don’t appear.   
distractor_agreement_relative_clause 22 63 60 68 86 Boys that aren’t disturbing Natalie sufferBoys that aren’t disturbing Natalie suffers  
irregular_plural_subject_verb_agreement_1 73 81 78 95 95 This goose isn’t bothering Edward. This goose weren’t bothering Edward.   
irregular_plural_subject_verb_agreement_2 88 89 83 96 94 The woman cleans every public park. The women cleans every public park.   
regular_plural_subject_verb_agreement_1 76 89 73 97 95 Jeffrey hasn’t criticized Donald. Jeffrey haven’t criticized Donald.   
regular_plural_subject_verb_agreement_2 81 83 85 96 95 The dress crumplesThe dresses crumples  
PhenomenonUID5-gramLSTMTXLGPT-2HumanAcceptable ExampleUnacceptable Example1pfx2pfx
Anaphor agreement anaphor_gender_agreement 44 88 91 99 96 Katherine can’t help herselfKatherine can’t help himself  
anaphor_number_agreement 52 95 97 100 99 Many teenagers were helping themselvesMany teenagers were helping herself  
 
Argument structure animate_subject_passive 54 68 58 77 98 Amanda was respected by some waitressesAmanda was respected by some picture  
animate_subject_trans 72 79 70 80 87 Danielle visited Irene. The eye visited Irene.   
causative 51 65 54 68 82 Aaron breaks the glass. Aaron appeared the glass.   
drop_argument 68 79 67 84 90 The Lutherans couldn’t skate around. The Lutherans couldn’t disagree with.   
inchoative 89 72 81 90 95 A screen was fading. A screen was cleaning.   
intransitive 82 73 81 90 86 Some glaciers are vaporizing. Some glaciers are scaring.   
passive_1 71 65 76 89 99 Jeffrey’s sons are insulted by Tina’s supervisor. Jeffrey’s sons are smiled by Tina’s supervisor.   
passive_2 70 72 74 79 86 Most cashiers are disliked. Most cashiers are flirted.   
transitive 91 87 89 49 87 A lot of actresses’ nieces have toured that art gallery. A lot of actresses’ nieces have coped that art gallery.   
 
Binding principle_A_c_command 58 59 61 100 86 A lot of actresses that thought about Alice healed themselvesA lot of actresses that thought about Alice healed herself  
principle_A_case_1 100 100 100 96 98 Tara thinks that she sounded like Wayne. Tara thinks that herself sounded like Wayne.   
principle_A_case_2 49 87 95 73 96 Stacy imagines herself praising this actress. Stacy imagines herself praises this actress.   
principle_A_domain_1 95 98 99 99 95 Carlos said that Lori helped himCarlos said that Lori helped himself  
principle_A_domain_2 56 68 70 73 75 Mark imagines Erin might admire herselfMark imagines Erin might admire himself  
principle_A_domain_3 52 55 60 82 83 Nancy could say every guy hides himselfEvery guy could say Nancy hides himself  
principle_A_reconstruction 40 46 38 37 78 It’s herself who Karen criticized. It’s herself who criticized Karen.   
 
Control/ raising existential_there_object_raising 84 66 76 92 90 William has declared there to be no guests getting fired. William has obliged there to be no guests getting fired.   
existential_there_subject_raising 77 80 79 89 88 There was bound to be a fish escaping. There was unable to be a fish escaping.   
expletive_it_object_raising 72 63 72 58 86 Regina wanted it to be obvious that Maria thought about Anna. Regina forced it to be obvious that Maria thought about Anna.   
tough_vs_raising_1 33 34 45 72 75 Julia wasn’t fun to talk to. Julia wasn’t unlikely to talk to.   
tough_vs_raising_2 77 93 86 92 81 Rachel was apt to talk to Alicia. Rachel was exciting to talk to Alicia.   
 
Deter-miner-noun agr. determiner_noun_agreement_1 88 92 92 100 96 Craig explored that grocery storeCraig explored that grocery stores  
determiner_noun_agreement_2 86 92 81 93 95 Carl cures those horsesCarl cures that horses  
determiner_noun_agreement_irregular_1 85 82 88 94 92 Phillip was lifting this mousePhillip was lifting this mice  
determiner_noun_agreement_irregular_2 90 86 82 93 85 Those ladies walk through those oasesThose ladies walk through that oases  
determiner_noun_agreement_with_adj_1 50 86 78 90 96 Tracy praises those lucky guysTracy praises those lucky guys  
determiner_noun_agreement_with_adj_2 53 76 81 96 94 Some actors buy these gray booksSome actors buy this gray books  
determiner_noun_agreement_with_adj_irregular_1 55 83 77 88 85 This person shouldn’t criticize this upset childThis person shouldn’t criticize this upset children  
determiner_noun_agreement_with_adj_irregular_2 52 87 86 93 95 That adult has brought that purple octopusThat adult has brought those purple octopus  
 
Ellipsis ellipsis_n_bar_1 23 68 65 88 92 Brad passed one big museum and Eva passed several. Brad passed one museum and Eva passed several big.   
ellipsis_n_bar_2 50 67 89 86 78 Curtis’s boss discussed four sons and Andrew discussed five sick sons. Curtis’s boss discussed four happy sons and Andrew discussed five sick.   
 
Filler gap wh_questions_object_gap 53 79 61 84 85 Joel discovered the vase that Patricia might take. Joel discovered what Patricia might take the vase.   
wh_questions_subject_gap 82 92 83 95 98 Cheryl thought about some dog that upset Sandra. Cheryl thought about who some dog upset Sandra.   
wh_questions_subject_gap_long_distance 86 96 86 88 85 Bruce knows that person that Dawn likes that argued about a lot of guys. Bruce knows who that person that Dawn likes argued about a lot of guys.   
wh_vs_that_no_gap 83 97 86 97 97 Danielle finds out that many organizations have alarmed Chad. Danielle finds out who many organizations have alarmed Chad.   
wh_vs_that_no_gap_long_distance 81 97 91 94 92 Christina forgot that all plays that win worry Dana. Christina forgot who all plays that win worry Dana.   
wh_vs_that_with_gap 18 43 42 56 77 Nina has learned who most men sound like. Nina has learned that most men sound like.   
wh_vs_that_with_gap_long_distance 20 14 17 56 75 Martin did find out what every cashier that shouldn’t drink wore. Martin did find out that every cashier that shouldn’t drink wore.   
 
Irregular forms irregular_past_participle_adjectives 79 93 91 78 99 The forgotten newspaper article was bad. The forgot newspaper article was bad.   
irregular_past_participle_verbs 80 85 66 90 95 Edward hid the cats. Edward hidden the cats.   
 
Island effects adjunct_island 48 67 65 91 94 Who has Colleen aggravated before kissing Judy? Who has Colleen aggravated Judy before kissing?   
complex_NP__island 50 47 58 72 80 Who hadn’t some driver who would fire Jennifer’s colleague embarrassed? Who hadn’t Jennifer’s colleague embarrassed some driver who would fire?   
coordinate_structure_constraint_complex_left_branch 32 30 36 42 90 What lights could Spain sell and Andrea discover? What could Spain sell lights and Andrea discover?   
coordinate_structure_constraint_object_extraction 59 71 74 88 91 Who will Elizabeth and Gregory cure? Who will Elizabeth cure and Gregory?   
left_branch_island_echo_question 96 32 63 77 91 David would cure what snakeWhat would David cure snake  
left_branch_island_simple_question 57 36 36 82 99 Whose hat should Tonya wear? Whose should Tonya wear hat?   
sentential_subject_island 61 43 37 35 61 Who have many women’s touring Spain embarrassed. Who have many women’s touring embarrassed Spain.   
wh_island 56 47 20 77 73 What could Alan discover he has run around? What could Alan discover who has run around?   
 
NPI licensing matrix_question_npi_licensor_present 67 98 Should Monica ever grin? Monica should ever grin.   
npi_present_1 47 54 61 55 83 Even these trucks have often slowed. Even these trucks have ever slowed.   
npi_present_2 47 54 48 62 98 Many skateboards also roll. Many skateboards ever roll.   
only_npi_licensor_present 57 93 80 100 92 Only Bill would ever complain. Even Bill would ever complain.   
only_npi_scope 30 36 45 85 72 Only those doctors who Karla respects ever conceal many snakes. Those doctors who only Karla respects ever conceal many snakes.   
sentential_negation_npi_licensor_present 93 100 99 89 93 Those banks had not ever lied. Those banks had really ever lied.   
sentential_negation_npi_scope 45 23 53 95 81 Those turtles that are boring April could not ever break those couches. Those turtles that are not boring April could ever break those couches.   
 
Quantifiers existential_there_quantifiers_1 91 96 94 99 94 There aren’t many lights darkening. There aren’t all lights darkening.   
existential_there_quantifiers_2 62 16 14 24 76 Each book is there disturbing Margaret. There is each book disturbing Margaret.   
superlative_quantifiers_1 45 63 84 84 91 No man has revealed more than five forks. No man has revealed at least five forks.   
superlative_quantifiers_2 17 83 85 78 85 An actor arrived at at most six lakes. No actor arrived at at most six lakes.   
 
Subject-verb agr. distractor_agreement_relational_noun 24 76 77 83 81 A sketch of lights doesn’t appear. A sketch of lights don’t appear.   
distractor_agreement_relative_clause 22 63 60 68 86 Boys that aren’t disturbing Natalie sufferBoys that aren’t disturbing Natalie suffers  
irregular_plural_subject_verb_agreement_1 73 81 78 95 95 This goose isn’t bothering Edward. This goose weren’t bothering Edward.   
irregular_plural_subject_verb_agreement_2 88 89 83 96 94 The woman cleans every public park. The women cleans every public park.   
regular_plural_subject_verb_agreement_1 76 89 73 97 95 Jeffrey hasn’t criticized Donald. Jeffrey haven’t criticized Donald.   
regular_plural_subject_verb_agreement_2 81 83 85 96 95 The dress crumplesThe dresses crumples  
Close Modal

or Create an Account

Close Modal
Close Modal