ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation

Compositional generalization benchmarks have emerged as powerful tools for diagnosing whether models have learned to systematically interpret natural language (Lake and Baroni, 2018; Kim and Linzen, 2020; Keysers et al., 2020; Ruis et al., 2020; Wu et al., 2021). The core task is to map sentences to logical forms (LFs), and the goal is to make accurate predictions for held-out examples that include novel grammatical combinations of elements. These tasks are guided by the assumption that natural languages are governed by the principle of compositionality: The meanings of complex phrases are determined by the meanings of their parts and the way those parts combine syntactically, and so the meanings of novel combinations of familiar elements are fully determined as well (Montague and Thomason, 1974; Partee, 1984; Janssen and Partee, 1997).

The COGS (COmpositional Generalization Challenge based on Semantic Interpretation) benchmark of Kim and Linzen (2020) is among the most widely used compositional generalization benchmarks at present, and it is noteworthy for containing assessment splits that almost no present-day models get traction on (Table 1). The phenomena are remarkably simple. For instance, in the COGS Object-to-Subject modification split (Table 2), models are trained on modified nouns like the cake on the plate only in grammatical object positions and asked to make predictions about such phrases when they are grammatical subjects. The average score for this split in prior work is roughly 0. Similar patterns obtain for splits requiring generalization to deeper clausal embedding. This looks like strong evidence that present-day models cannot handle basic matters of semantic interpretation.

We argue that this conclusion is hasty. The core issue is one of semantic representation. The goal of COGS is to assess whether models can compute meanings compositionally, but this is operationalized as the task of predicting LFs in a specific format. There will in general be innumerable LF formats that express the desired meanings, and we have no reason to privilege any particular one.

In this paper, we focus on three incidental details of syntactic form that profoundly impact model performance and thus challenge the validity of COGS. First, we find that when we make trivial, meaning-preserving modifications to the LFs by removing redundant symbols, we see substantial improvements in model performance (Section 4.2).

We also identify subtler issues related to variable binding. In COGS, all variables are bound.¹ They appear unbound in the LFs but they are interpreted as existentially closed, as in many theories of dynamic semantics (Kamp, 1981; Heim, 1982, 1983). As bound variables, they can be freely renamed with no change to the interpretation as long as the renaming is consistent; truck(x) and truck(y) are semantically identical in COGS because both variables are implicitly bound with existentials. However, COGS currently requires models to predict the exact identity of variables. This goes well beyond capturing semantics. For neural models that rely on token embeddings, this poses an artificial challenge for test LFs that happen to contain novel variable names or familiar variable names that happen to appear in new contexts. This affects COGS splits involving novel clausal embedding depths (Section 4.3) and novel modification patterns (Section 4.4). Again, meaning-preserving adjustments to the COGS LFs address these issues and allow even baseline models to succeed.

At the same time, we emphasize that COGS does already contain instances of variable binding relationships that are challenging for all present-day models. A recent proposal by Qiu et al. (2022) to map all COGS LFs to variable-free forms is not meaning preserving precisely because it cannot handle these binding relationships properly (Section 3). Thus, the high scores models have obtained for this COGS variant are partly illusory.

We close with a proposal for a revised version of COGS, ReCOGS, that incorporates the above insights (Section 5). ReCOGS is easier than COGS in some respects and harder in others. Through ablation studies in Section 5, we show that the original linear variable binding of COGS actually make some aspects of COGS artificially easier, and can prevent us from accurately accessing compositional generalization of models. As we noted above, any particular choice of LFs will be somewhat arbitrary relative to our goals, but we feel that ReCOGS comes closer to assessing whether our models possess the compositional generalization capabilities that Kim and Linzen (2020) identified as essential, and our findings suggest that present-day models continue to struggle in these areas.²

COGS consists of input–output pairs mapping English sentences to LFs. The dataset is generated using a rule-based approach, which allows COGS to maintain systematic gaps between training and different evaluation splits. COGS LFs are based on a Neo-Davidsonian view of verbal arguments (Parsons, 1990), in which verbs introduce event arguments and participants are linked to those events via thematic role predicates. As discussed in the original paper (Kim and Linzen, 2020), the LFs are created by post-processing the simplified ones defined in Reddy et al. (2017). The LFs are purely conjunctive (conjunction is denoted by ; and AND), and conjuncts are sorted by their variable names, which are determined by the position of the head phrase in the sentence (see Table 2 for examples). Event predicates for nominals are not included. Definite and indefinite phrases are formally distinguished. All COGS variables are bound; the definiteness operator * binds variables locally to its conjunct, and all other variables are interpreted as bound by implicit widest-scope existential quantifiers.

By convention, the subscripts on variables in COGS correspond to the 0-indexed position of the corresponding word in the input sentence. Thus, the LF for A cat rolled Lina includes a conjunct cat(x_1), which indicates cat is the first word in the sentence, whereas the LF for Lina rolled a cat includes a conjunct cat(x_3), which indicates cat is the third word in the sentence.

The COGS evaluation metric is percent exact string identity of logical forms. COGS provides a single training split as well as standard in-distribution (IID) validation splits. In addition, the dataset includes generalization splits designed around types of examples that were not seen in training but seem to be natural extrapolations of examples seen in training under assumptions of compositionality, as exemplified in Table 2. The generalization splits cover five scenarios:

1. Interpreting novel pairings of primitives and grammatical roles (e.g., Subj → Obj Proper).

2. Verb argument structure alternation (e.g., Active → Passive).

3. Sensitivity to verb class (e.g., Agent NP → Unaccusative subject).

4. Interpreting novel combinations of modified phrases and grammatical roles (e.g., Object PP → Subject PP).

5. Generalizing phrase nesting to unseen depths (e.g., CP Recursion).

The first three fall under lexical generalization, and the final two require structural generalization.

It is noteworthy that, for the splits in group 1, there is only a single training example for the primitive with a single input and output token (e.g., “Paula” →Paula). This leads to higher variance across these related splits; we suspect that model performance for these cases is affected by when this single example is seen by the model during training.

Researchers have adopted a variety of approaches to solving COGS, including grammar-based rules (Herzig and Berant, 2021), lexicons or lexicon-style alignments incorporated into seq2seq models (Akyurek and Andreas, 2021; Zheng and Lapata, 2021), modified Transformer models for better- structured representations (Oren et al., 2020; Zheng and Lapata, 2022; Bergen et al., 2021; Csordás et al., 2021), meta-learning (Conklin et al., 2021), tree-like neural parsers (Weißenhorn et al., 2022), a grammar-enhanced seq2seq learner (Wang et al., 2022), and various data augmentation techniques (Qiu et al., 2022).

A number of recent papers investigate artifacts in compositional generalization benchmarks such as SCAN (Patel et al., 2022) and ReaSCAN (Sikarwar et al., 2022). Csordás et al. (2021) focus on COGS. They examine potential pitfalls that might lead us to underestimate a model’s ability to generalize. By carefully exploring the effects of relative positional embeddings and training time, they are able to increase overall model performance from 35% to 81% on the generalization splits. Our work complements these findings by focusing on issues of semantic representation.

Qiu et al. (2022) propose a variable-free version of COGS and show that models score much better on the format (which is used by Drozdov et al. (2023) in experiments with GPT-3). However, the representations of Qiu et al. do not preserve the meaning of COGS examples. COGS embeds variable-binding challenges that the variable-free forms artificially side-step. For instance, their formalism represents A zebra needs to walk as need(agent=zebra,xcomp=walk(agent=zebra)) which means ‘a zebra needs a zebra to walk’, with two unlinked occurrences of the indefinite a zebra. Consider a model where zebra a needs zebra b to walk, for a ≠ b, and no zebra a is such that a needs a to walk. The above LF is true in this model but our original sentence is not (McCawley, 1968). In addition, these variable-free forms would be unable to represent quantifier-binding relationships like Every zebra ate its meal as well as simple reflexives like A zebra saw itself. Since variable binding arguably reflects one of the deepest challenges of natural language interpretation, we argue that these phenomena should not be marginalized.³

We report on experiments studying semantic representations for COGS. For any modifications we apply to the dataset, we ensure there is no data leakage, as such leakage trivializes the generalization tests (as evidenced by the consistently very high results for COGS IID test splits in prior work).

Following the original COGS paper (Kim and Linzen, 2020), we train encoder–decoder models with two model architectures: LSTMs (Hochreiter and Schmidhuber, 1997) and Transformers (Vaswani et al., 2017). We adopt similar configurations to the original paper. For the LSTMs, we use a 2-layer LSTM as the encoder with global attention and a dot-product scoring function, and a 2-layer LSTM as the decoder with a hidden dimension size of 512. For the Transformers, we use two 2-layer Transformer blocks with 4 attention heads and learnable absolute positional embeddings⁴ with a hidden dimension size of 300.

The LSTM architecture has approximately 9M parameters whereas the Transformer architecture has approximately 4M parameters.

We use cross-entropy loss and a fixed number of training epochs (200 for LSTMs and 300 for Transformers), as previous work suggests that early stopping hurts model performance (Csordás et al., 2021). We set the batch size to 128 for the Transformers and 512 for the LSTMs. We train with a fixed learning rate of 8× 10⁻⁴ for the LSTMs, and 1× 10⁻⁴ for the Transformers.

We use a single NVIDIA GeForce RTX 3090 24GB GPU to train our models. For LSTMs, the training time ranges from 0.5 hours to 5 hours. For Transformers, it ranges from 0.3 hours to 3 hours. We run each experiment with 20 distinct random seeds. Additional training details can be found in our code repository.

We train all of our models from scratch, without any pretraining, to ensure that we are not introducing outside information that could be relevant to the COGS generalization tasks (q.v. Kim et al., 2022).

Our first experiments involve only very trivial modifications to COGS LFs: We remove some redundant tokens from the LFs, with no other modifications to the examples, and we study the effects this has on the lexical generalization splits in COGS. (These modifications alone do not significantly improve results on the structural generalization splits.)

Table 3 summarizes our redundant token removal strategies. The first row gives a COGS example. Each noun or verb is associated with a variable representing its token position in the input sentence. Concretely, these variables are given as three whitespace-separated elements: x _ N, where N is a numeral. The initial x, the underscore, and the spaces do not contribute to variable identity in any way. Thus, we can remove these prefixes without changing the semantics of the LFs. The second row of Table 3 depicts removal of the underscore, and the third row depicts removal of the entire prefix. Additionally, we experiment with different token removal schemes by removing redundant punctuation in the set {,()}.

Table 4 illustrates model performance on the three most demanding lexical generalization splits when models are trained and assessed with these minor variants of COGS LFs. Our primary finding is that model performance is highly sensitive to redundant token removal. By removing the prefix x _ , the Transformer achieves nearly 30% better performance on average for the Primitive → Object Proper Noun split, and the LSTM’s performance improves 27% for the Primitive → Subject Proper Noun split after removing the prefix x _ and parenthesis tokens.

Our findings show that model performance can vary substantially across different semantically equivalent LFs. In addition, while model performance is stable in the Overall evaluation, some of the splits show a high degree of sensitivity to the random seed. Nonetheless, even in these splits, the performance increases that stem from redundant token removal are consistently large enough to be robust to this variance.

We hypothesize that the improvements that come from removing these redundant tokens derive from simple considerations of how sequence models operate. As shown in Figure 2, the bigram , x appears 44,846 times, whereas the bigram , Emma appears 4,279 times (Emma is the most frequent proper noun in the training split). Thus, the conditional bigram probability P(x∣,) is significantly larger than P(Emma∣,). Removing the prefix balances the dataset in this respect because we then mostly care about P(N∣,) for different values of N. This helps the decoder to generate less skewed label distributions. This corroborates the findings of Yao and Koller (2022), who show that the decoder is heavily biased towards generating seen n-grams.

The splits that focus on recursive phrase embedding are among the hardest problems in COGS. When training, models see multiply embedded examples like [John knows that [Noah knows that …]_CP]_CP containing nestings of depth 0–2. At test time, they see examples involving strictly greater depth (3–12). COGS includes two constructions that allow nesting: sentential complements (nested CPs) and nominal PP modifiers (nested PPs).

The nature of these COGS splits creates a strict relationship between recursion depth and sequence length: deeper recursion leads to longer sequences. Figure 3 summarizes the situation when it comes to input sentences and output logical forms. As expected, we see non-overlapping long-tail distributions over longer sequences.

This is perfectly well-posed as a generalization task simultaneously assessing models on both longer lengths and deeper recursion. However, COGS binds these two tasks together in a way that takes us outside of the goal of assessing semantic generalization. Recall that COGS variables are named according to their linear position. This virtually guarantees that there will be variables whose numerical components are encountered only at test time. For instance, if 47 is the largest variable index seen at train time, then 48 and above will be encountered only in testing. For models that rely on embeddings, these new variables will have random representations at test time. In other words, COGS is underspecified in a way that prevents models from learning novel positional embeddings or embeddings for novel positional indices. However, there is nothing privileged about this particular variable naming scheme; as we discussed earlier, bound variables can be freely renamed as long as this does not change any binding relationships.

Relatedly, for models like Transformers with token positional embeddings, some of the position representations will be encountered only at test time. It is fair to say that this is a limitation of these models that COGS is exposing, but it also further reinforces the concern that length generalization may be totally overshadowing the challenge of generalizing to novel recursion depths.

To overcome the challenge of length generalization, we augment the training data by concatenating existing examples together, reindexing the output LFs with higher positional indexes, and gluing them together as conjunctive terms. In doing this, we do not add any semantically new claims to the COGS training set, and we do not create any new recursive structures. We simply ensure that the relevant token and position indices are not random at test time. Similar concatenation experiments have been shown effective in understanding model artifacts for language model acceptability tasks (Sinha et al., 2023).

Figure 4 presents our results. We gradually introduce k augmented examples into our training set, where k ∈{256,512,1024,2048,3072}. Our results indicate that the previously seen catastrophic failure of structural generalization over nested clauses is largely due to the fact that models are not trained with longer sequences. When the sequence length issue is addressed, the models appear to be very capable at handling novel recursion depths. Indeed, our models now far surpass published state-of-the-art results on these tasks.

We further note that the failures are not due to our setup of relative positional embeddings, as fine-tuning a T5-base model with fixed window-based relative positional embeddings remains 0.0% on both splits, as shown in Table 1.

The hardest COGS split based on published numbers seems to be the structural generalization task that involves interpreting novel combinations of modified phrases and grammatical role—e.g., interpreting subject noun phrases with PP modifiers when the train set includes only object noun phrases with such modifiers (Noah ate the cake on the plate.→The cake on the plate burned). To the best of our knowledge, all prior seq2seq models have completely failed to get traction here (Table 1). Our goal in this section is to understand more deeply why this split has proven recalcitrant.

To start, we observe that this split is arguably different conceptually from the others. The train set contains only object-modifying PPs. It is quite reasonable for a learner to infer from this situation that PPs are allowed only in this position. Natural languages manifest a wide range of subject/object asymmetries, and learners presumably induce these at least in part from an absence of certain kinds of inputs in their experience. Thus, there is a case to be made that this split is not strictly speaking fair in the sense of Geiger et al. (2019): we have a generalization target in mind as analysts, but this target is not uniquely defined by the available data in a way that would invariably lead even an ideal learner to the desired conclusion.

That said, we feel it is reasonable to explore whether models can learn the sort of theory that would naturally allow them to make correct predictions about these particular novel inputs, even if their training experiences run counter to that. However, COGS erects another obstacle to this goal by numbering variables using a word’s position index in the sentence. At train time, the model sees only PP modifiers like the one in the first row of Table 3. This associates PP modifiers with a particular range of tokens (variable numerals). As before, this indexing scheme is not a semantic matter, but rather a superficial matter of representation. However, it can also be seen as very direct supervision about the limited distribution of these modifiers: They can associate only with relatively high variable indices.

Additionally, for models with learned positional indices, all modifier phrase tokens are associated with a particular set of relatively high positional index values. These models thus further reinforce the distributional inference that such phrases cannot appear in subjects.

Both of the above concerns are supported by error analyses on vanilla models trained on the original COGS LFs. For instance, to correctly generate the LF for subject-modifying PPs, the model has to predict the LF term for a PP modifier as the first conjunctive clause after the semicolon (e.g, cake .nmod .on). However, 0% of the model’s predictions follow this pattern, as this never happens during training. Likewise, since no subject-modifying PPs are seen in training, the variables mentioned in PP clauses are never associated with the subject. The effect of these patterns on model behavior is clear: the first variable of any generated modified phrase clause always refers to a token position seen during training.

To isolate these effects from the structural generalization, we propose three data modification strategies. None of these strategies change the set of meanings expressed by COGS. Rather, they make adjustments to the syntax that, according to the COGS indexing rules, automatically expand the range of variables and positional indices associated with modifier phrases. Examples for each strategy are given in Table 5.

One strategy to disentangle the effects of variable names and positional indices is to move the modified phrases to the front of the sentence via preposing (topicalization). This has no effect on argument structure or the meaning of the corresponding LF. It simply ensures that the model sees concurrences of positional indices that would otherwise have only appeared in the testing set. In our experiments, we prepose the object in 5% of training examples containing at least one prepositional phrase.

Preposing brings a new challenge for parsing, since the model needs to learn how to parse preposed PP modifiers for object NPs as well as the regular ones. Despite this new challenge, preposing dramatically improves model performance.

To further alleviate the problem with unseen occurrences of positional indices, we experiment with adding filler words (“um”) into the input sentence. This shifts around the variable names and positional indices in the LF (see the corresponding example in Table 3). We sprinkle 1–3 filler words at random in 5% of training examples with one or more prepositional phrase. Filler words can be seen as mapping to the constant conjunct with meaning True, and thus they do not affect the overall meaning of the LF.

We augment the training set by providing examples for participial verb (PV) phrases for both objects and subjects. The original vocabulary of COGS does not have any participial verbs, so we added participial forms for all verbs.⁵ To create full phrases, we randomly select a participial verb and a noun. Then, we randomly assign an article for the noun selected. For two participial verb phrases in a row, we repeat this process twice. The vocabulary files of the models are updated to include the new tokens required for parsing these PV phrases.

As shown in the corresponding row in Table 5, the model now needs to learn this new type of noun modifier acl with participial verbs. We argue that testing with PP modifiers within subject NPs becomes more reasonable in this new setting, as we completely isolate the potential artifacts due to the positional indices. We add PV phrases to 15% of training examples with maximally two PV phrases in a row. We include an easy version without adding the novel acl token and replace it with the existing nmod token in the LF.

As shown in Table 6, both the LSTM and the Transformer model start to show progress on this structural generalization split. Surprisingly, the LSTM model becomes much more effective in parsing subject NPs after including PV phrases, which contrasts with previous findings that the Transformer model is always better than the LSTM model at structural generalization on average. Our modifications isolate the effect of variable name bindings from structural generalization. The results suggest that the stagnation of model performance on this split is mostly due to the particular variable naming convention of the current semantic representation. We later show (in Section 5) that models overfit to the variable binding of COGS, which could make COGS artificially easier, and prevent us from accurately accessing compositional generalization of models.

We now propose a revision to COGS called ReCOGS. We assemble the insights provided by the above experiments and use them to inform a benchmark that comes closer to assessing models purely on their ability to handle semantic generalizations. Table 7 provides an example. We first implement the following changes, supported by our experiments, to create ReCOGS:

1. We remove redundant prefix tokens including x and _ (motivated by Section 4.2).

2. We replace each token position index with a random integer available in the model vocabulary file, maintaining consistent coreference. For instance, every appearance of x_3 may be replaced with x_46, which makes the indices irrelevant to their token positions in the sentence. For each original COGS example, we create 5 different versions by randomly sampling 5 distinct sets of indices. We provide a ReCOGS variant, ReCOGS_POS, without this change to show its effect.

3. We augment the current training split with longer examples by concatenating existing training examples. In total, we add in 15,360 (3,072 for each set of indices) new examples. We make sure only unique examples are kept (motivated by Section 4.3).

4. We prepose the object for 5% of the training examples containing an object with at least one prepositional phrase, and we randomly add filler words (“um”) into these examples (motivated by Section 4.4).

Additionally, we make the following changes that are not experimentally supported but help make ReCOGS LFs more consistent and expressive:

1. We treat proper nouns as predicates, which prevents collisions when multiple distinct entities share the same name (e.g., Mia(4) and Mia(5) can refer to different people). This also leads to a more uniform semantic treatment of noun phrases in general.

2. We prepose proper nouns and nouns with indefinite articles in the LF, parallel to definites. This makes LFs more consistent.

3. We separate situations and semantic roles, and rely on variable binding to link them (e.g., reformat agent.eat(6,7) as eat(6) AND agent(6,7)). The resulting LFs more closely resemble the Neo-Davidsonian view of verbal arguments (Parsons, 1990).

Table 8 provides basic dataset statistics for COGS and ReCOGS.

We emphasize that ReCOGS continues to use variables in its representations. As we noted in Section 3, the variable-free proposal of Qiu et al. (2022) does not always preserve the meanings of the original COGS representations. COGS embeds variable-binding challenges that this variable-free form artificially side-steps, something that we feel is inappropriate when variable binding arguably reflects one of the deepest challenges of natural language interpretation.

As a metric for ReCOGS, we propose Semantic Exact Match (SEM). For SEM, we exhaustively check whether there is a semantically consistent conversion of the variables in the predicted LFs into those of the actual LF. For COGS, this amounts to checking whether there is bijective map between variables in the predicted and actual LFs, where corresponding variables participate in exactly the same predications.⁶ For instance, the predicted LF table(46) AND sturdy(46) can be converted to table(1) AND sturdy(1) via the mapping [46↦1], but the LF table(46) AND sturdy(7) cannot be converted in this way because 46 and 7 would have to both map to 1. The reverse conversion is also blocked because 1 would have to map to both 46 and 7. In addition, we treat conjunctions in the LF as an order-free set, and exact match is evaluated as set equality after variable conversion. For the COGS language, these steps amount to checking for semantic equivalence.⁷

We evaluate ReCOGS with the same two model architecture setups as in previous sections. Figure 5 reports initial results. Overall, we find that ReCOGS is a more challenging compositional generalization benchmark than COGS, but models are able to get traction on all splits, suggesting we have avoided the worrisome pattern of 0s seen in Table 1.

While both our models still achieve near 100% performance on the IID testing split, they struggle on the structural and lexical generalization tests. However, they now show signs of life. For instance, the Object PP → Subject PP split of ReCOGS is now tractable while remaining challenging. This shows how the meaning-preserving modifications of “um” insertion and preposing can remove artifacts that were preventing models from learning before. In addition, our results show that LSTMs perform better than Transformers on the same split (13.4% vs. 4.5% on average). Future work may investigate the reason why LSTMs seem to generalize better in this scenario.

We also see the effects of breaking the relationship between variable names and linear position. Without this non-semantic pattern to rely on, models show degraded performance. To verify this, we revert ReCOGS back to the original variable name bindings of COGS where the order of LFs mirrors the word order in the input sequence, and generate a new variant, ReCOGS_POS. As shown in Figure 5, performance of both models increases significantly on ReCOGS_POS. Consequently, we believe that ReCOGS poses additional challenges in terms of long-term coindexing over entities, challenges that were partly obscured by the variable naming scheme chosen for COGS.

With compositional generalization benchmarks, we hope to gain reliable information about whether models have learned to construct the meanings of natural language sentences. However, meanings are highly abstract entities that we do not know how to specify directly, so we are compelled to conduct these assessments using LFs, which are syntactic objects that we presume can themselves be mapped to meanings in the relevant sense. The central difficulty here is that there is no single privileged class of LFs to use for this purpose, and different LFs are likely to pose substantially different learning problems and thus may lead to very different conclusions about our guiding question.

We explored these issues in the context of COGS, a prominent compositional generalization benchmark. COGS includes sub-tasks that even our best present-day models cannot get any traction on. Our central finding is that there are two major factors contributing to this abysmal performance: redundant symbols in COGS LFs and the requirement imposed by COGS that models predict the exact numerical values of bound variables. These details cannot be justified semantically, and they play a large role in shaping model performance. We showed that simple, meaning-preserving modifications to COGS fully address these problems and allow models to succeed. In turn, we propose ReCOGS, a modified version of COGS that incorporates these insights. Our models are able to get some traction on all the sub-tasks within ReCOGS, but it remains a very challenging benchmark.

One of our reviewers raised an important question relating to ReCOGS and benchmarking efforts that might follow from it: Could this encourage an unproductive sort of “LF hacking” in which people continually reformat the data in an effort to incrementally boost performance? This is certainly a risk. However, for now, we venture that such experimentation can be productive. If the modified LFs are truth-conditionally identical to the originals, we can be confident that the generalization splits are not being compromised by this LF hacking. Thus, consistent improvements in performance may lead to lasting insights about effective meaning representation for our models, and persistent failures are likely to point to deep limitations. Overall, then, some LF hacking could help us get incrementally closer to truly evaluating the capacity of models to compute meaning.

Our results also raise important questions about compositional generalization itself. It is in the nature of compositional generalization tasks that models will be assessed on examples that are meaningfully different from those they have seen in training. When is this fair, and when does it implicitly contradict the train set itself? Our discussion focused on the distribution of PP modifiers across different grammatical positions. COGS models are trained on examples that might suggest these are limited to object noun phrases. This may seem implausible for PP modifiers, but many natural languages do display subject/object asymmetries with regard to phenomena like case marking, definiteness, and pronominalization. In our experiments, we indirectly instructed our model about our intended generalization via meaning preserving modifications to the training data, but ultimately these issues may call for deeper changes to how we pose compositional generalization tasks.

We would like to thank Benjamin Van Durme and anonymous reviewers for their helpful suggestions. We are also grateful to Najoung Kim, Bingzhi Li, Tal Linzen, Yuekun Yao, and Linlu Qiu for very helpful discussion.