The Causal Influence of Grammatical Gender on Distributional Semantics

Approximately half of the world’s languages have grammatical gender (Corbett, 2013a), a grammatical phenomenon that groups nouns together into classes that share morphosyntactic properties (Hockett, 1958; Corbett, 1991; Kramer, 2015). Among languages that have gender, there is variation in the number of gender classes; for example, some languages have only two classes, e.g., all Danish nouns are classed as either common or neuter, whereas others have significantly more, e.g., Nigerian Fula has around 20, depending on the variety (Arnott, 1967; Koval’, 1979; Breedveld, 1995). Languages also vary with respect to how much gender assignment, i.e., how nouns are sorted into particular genders, is related to the form and the meaning of the noun (Corbett, 1991; Plaster and Polinsky, 2007; Corbett, 2013b, 2015; Kramer, 2020; Sahai and Sharma, 2021). Some languages group nouns into gender classes that are highly predictable from phonological (Parker and Hayward, 1985; Corbett, 1991, 2013b) or morphological (Corbett, 1991, 2013b; Corbett and Fraser, 2000) information, while others, such as the Dagestanian languages Godoberi and Bagwalal, seem to be predictable from meaning (Corbett, 1991; Corbett and Fraser, 2000; Corbett, 2015)—although, even for most of the strictly semantic systems, there are exceptions.

Despite this variation, gender assignment is rarely, if ever, wholly predictable from meaning alone. In many languages, there is a semantic core of nouns that are conceptually coherent (Aksenov, 1984; Corbett, 1991; Williams et al., 2019; Kramer, 2020) and a surround that is somewhat less semantically coherent. Axes along which genders are conceptually coherent often include semantic properties of animate nouns, with inanimate nouns appearing in the surround. For example, in Spanish, despite the fact that the nouns table (mesa in Spanish) and woman (mujer in Spanish) appear in the same gender (i.e., feminine), it is hard to imagine what meaning they share. Indeed, some linguists posit that gender assignment for inanimate nouns is effectively arbitrary (Bloomfield, 1935; Aikhenvald, 2000; Foundalis, 2002). And, to the extent that gender assignment is not fully arbitrary for inanimate nouns (Williams et al., 2021), many researchers argue there is no compelling evidence showing grammatical gender affects how we conceptualize objects (Samuel et al., 2019) or the distributional properties of language (Mickan et al., 2014).

However, not all researchers agree that non- arbitrariness in gender assignment, to the extent it exists, should be assumed to have no bearing on language production. Boroditsky (2003) famously argued for a causal relationship between the gender assigned to inanimate nouns and their usage, in a view colloquially known as the neo-Whorfian hypothesis after Benjamin Whorf (Whorf, 1956). Proponents of this view have studied human associations, under the assumption that people’s perceptions of the genders of objects are strongly influenced by the grammatical genders these objects are assigned in their native language (Boroditsky and Schmidt, 2000; Semenuks et al., 2017). One manifestation of this perception is the choice of adjectives used to describe nouns (Semenuks et al., 2017). While this is an intriguing possibility, there are additional lexical properties of nouns that may act as confounders and, thus, finding statistical evidence for the causal effect of grammatical gender on adjective choice requires great care.

To facilitate a cleaner way to reason about the causal influence grammatical gender may have on adjective usage, we introduce a causal graphical model to represent the interactions between an inanimate noun’s grammatical gender, its meaning, and the choice of its descriptors. This causal framework enables intervening on the values of specific factors to isolate the effects between various properties of languages. Our model explains the distribution of adjectives that modify a noun, conditioned on both a representation of the noun’s meaning and the gender of the noun itself. Upon estimation of the parameters of the causal graphical model, we test the neo-Whorfian hypothesis beyond the anecdotal level. First, we validate our model by comparing it to the method presented in prior work without any causal intervention. Second, we employ our model with a causal intervention on the noun meaning to test the neo-Whorfian hypothesis. That is, we ask a counterfactual question: Had nouns been lexicalized with different grammatical genders but retained their same meanings, would the distribution of adjectives that speakers use to modify them have been different?

We employ our model on five languages that exhibit grammatical gender: four Indo-European languages (German, Polish, Portuguese, and Spanish) and one language from the Afro-Asiatic language family (Hebrew). We find that, at least in Wikipedia data, a noun’s grammatical gender is indeed correlated with the choice of its descriptors. However, when controlling for a confounder, noun meaning, we present empirical evidence that noun gender has no significant effect on adjective usage.

In many languages with grammatical gender, adjectives, demonstratives, determiners, and other categories agree with the noun in gender, i.e., they will systematically change in form to indicate the grammatical gender of the noun they modify. Observe the following sentence, A small dog sleeps under the tree., translated into two languages that exhibit grammatical gender (German and Polish):

• Ein kleiner Hund schläft unter dem Baum. (de)

  a.M small.M dog.M sleeps under the.M tree.M

• Mały pies śpi pod drzewem. (pl)

  a.M small.M dog.M sleeps under the.N tree.N

Because the German (de) and Polish (pl) words for a dog, Hund and pies, are both assigned masculine gender, the adjectives in the respective languages, klein and mały, are morphologically gender-marked as masculine. Additionally, in German, the article, dem, is also gender-marked as masculine. The fact that gender is reflected by agreement patterns on other elements is generally taken to be a definitional property (Hockett, 1958; Corbett, 1991; Kramer, 2020) separating gender from other kinds of noun classification systems, such as numeral classifiers or declension classes.

It is an undeniable fact in many languages that morphological agreement reflects the gender of a noun in the form of other elements. However, one could imagine a similar process, such as analogical reasoning (Lucy, 2016), by which gender could influence both a noun’s meaning and its form. If a noun’s meaning were to influence its gender, then the noun meaning could also indirectly influence adjective usage, by way of the relationship between grammatical gender and adjective usage. There is ample statistical evidence that grammatical gender assignment is not fully arbitrary (Williams et al., 2019, 2021; Nelson, 2005; Sahai and Sharma, 2021). Such evidence is prima facie consistent with the idea that such influence is conceivably possible.

However, it is important to note that claims that noun gender influences meaning are by their very nature causal claims. The most famous example of such a causal claim is the neo- Whorfian view of gender (Boroditsky and Schmidt, 2000; Boroditsky, 2003; Semenuks et al., 2017), which states that a noun’s grammatical gender causally affects meaning (e.g., adjective choice). Boroditsky and Schmidt (2000) discuss a laboratory experiment they have conducted showing that speakers of German chose stereotypically feminine adjectives to describe, for example, bridges, while speakers of Spanish chose stereotypically masculine adjectives. Boroditsky and Schmidt concluded that participant adjective choice reflected the fact that in German, the word for a bridge, Brücke, is grammatically feminine, while in Spanish, the word for a bridge, puente, is grammatically masculine. Their findings can be summed up in the following quote from Boroditsky and Schmidt (2000), “people’s ideas about the genders of objects are strongly influenced by the grammatical genders assigned to these objects in their native language” (emphasis ours). Despite this clear causal formulation of the hypothesis, there has yet to be a modeling approach developed to test it. Moreover, subsequent studies have failed to replicate this result, raising into question the strength of this relationship between gender and adjective usage (Mickan et al., 2014) and inviting a study with more appropriate methodology.

Our paper builds on Williams et al.’s (2021) correlational study of noun meaning and its distributional properties and advances it to a causal one. While Williams et al. (2021) report non-trivial, statistically significant mutual information between the grammatical gender of a noun and its modifiers, e.g., adjectives that modify the noun, they do not control for other factors which might influence adjective usage, most notably the lexical semantics of the noun. Mutual information is symmetric and thus cannot speak to causation on its own. We are thus motivated by a potential common cause whereby the lexical semantics jointly influence a noun’s grammatical gender and its distribution over modifiers and propose a causal model.

The technical contribution of this work is a novel causal graphical model that jointly represents the relationship between the grammatical gender of a noun, its meaning, and descriptors. This model is depicted in Figure 1. If properly estimated, the model should enable us to measure the causal effect of grammatical gender on adjective choice in language. We first develop the necessary notation.

We follow several font and coloring conventions to make our notation easier to digest. All base sets will be uppercase and in calligraphic font, e.g., $X$⁠. Elements of $X$ will be lowercase and italicized, e.g., $x∈X$⁠. Subsets (including submultisets) will be uppercase and unitalicized, e.g., $X⊂X$⁠. Random variables that draw their values from $X$ will be uppercase and italicized, e.g., p(X = x). We will use three colors. Those objects that relate to nouns will be in blue, those objects that relate to adjectives will be in purple, and those objects that relate to gender will be in green.

We assume there exists a set of noun meanings $N$⁠. In this paper, we assume that such meanings are representable by column vectors in ℝ^D. We denote the elements of $N$ as n ∈ℝ^D. Additionally, we assume there exists an alphabet of adjectives $A$⁠. We denote an element of $A$ as a. Finally, we assume there exists a language-dependent set of genders $G$⁠. In Spanish, for instance, we would have $G={fem,msc}$ whereas in German $G={fem,msc,neu}$⁠. We denote elements of $G$ as g.

Importantly, our model generates the lexical semantics of noun types. This means that a sample from it generates a new noun, whose semantics we may never have seen before. If we are able to estimate such a model well, we can use the basics of causal inference to estimate the causal effect gender has on adjective usage. Specifically, as is clear from Figure 1, the only confounder between gender and adjective selection in our proposed model is the semantics of the noun.¹

Relating the weighted Jensen–Shannon divergence to a specific mutual information provides a clear interpretation. This measure explains in bits how much the entropy of the language’s distribution over adjectives is reduced when the grammatical gender of the noun being modified is known at the time of the adjective choice. For instance, if the language’s distribution over adjectives has an entropy H(A) of 10 bits and the mutual information $MI(A;G)=defH(A)−H(A∣G)$ is 1 bit, then knowing the gender allows us to reduce the uncertainty over which adjectives modify the nouns to H(A∣G) = 9 bits. However, the reduced uncertainty measured by MI(A;G) is purely associational; we cannot conclude that the gender of the noun actually causes the change in adjective distribution. Such a change could also be attributed to a confounding factor (like noun meaning). On the other hand, $MIdo(A;G)= defHdo(A)−Hdo(A∣G)$ represents the amount of uncertainty in the adjective distribution causally reduced by the gender random variable. Intuitively, we can reason about H_do(A) and H_do(A∣G) as the uncertainty of the adjective distribution in a world where we can counterfactually imagine that all nouns have the same gender g, and thus by setting all else equal, isolate the effect of knowing gender alone on the uncertainty of the adjective distribution. For a formal definition of H_do(A) and H_do(A∣G), see Appendix I.

We now explain the intuition behind the mutual information MI_do(A;G).

First, consider the case when there is no edge from N to G, indicating that there is no causal relationship between a noun’s meaning and its grammatical gender. Under this condition, we have $pG(g∣n)=pG(g)∀g∈G,n∈N$⁠. Consequently, the interaction between the grammatical gender of a noun and the adjectives used to describe this noun is not mitigated by the meaning of this noun, and thus, MI_do(A;G) = MI(A;G).

Second, consider the case when there is no edge from G to A, indicating that there is no causal relationship between a noun’s gender and its adjective distribution. In this case, MI_do(A;G) = 0. Further, we can show that MI_do(A;G) = 0 if and only if the edge from G to A does not exist.

Finally, when both edges from G to N and G to A exist, MI_do(A;G) can vary. In particular, MI_do(A;G) is non-negative (and indeed non-zero by case 2). However, we know of no relationship between MI_do(A;G) and MI(A;G). In this case, the strength of the relationship between a noun’s grammatical gender and adjective choice is regulated by the meaning.

We opt to model p_G(g∣n) and p_N(n) as the empirical distribution of nouns in the corpus.

In this section, we describe the data used in our experiments, and how we estimate non-contextual word representations as a proxy for a noun’s lexical semantics.

We gather data in five languages that exhibit grammatical gender agreement: German, Hebrew, Polish, Portuguese, and Spanish. This is certainly not a representative sample of the subset of the world’s languages that exhibit grammatical gender, but we are limited by the need for a large corpus to estimate a proxy for lexical meaning. Hebrew, Portuguese, and Spanish distinguish between two grammatical genders (masculine and feminine), while German and Polish distinguish between three genders (masculine, feminine, and neuter).³

We use the Wikipedia dump dated August 2022 to create a corpus for each of the five languages,⁴ and preprocess the corpora with the Stanza library (Qi et al., 2020).⁵ Specifically, we tokenize the raw text, dependency parse the tokenized text, lemmatize the data, extract lemmatized noun– adjective pairs based on an amod dependency label, and finally filter these pairs such that only those for inanimate nouns remain. To determine which nouns are inanimate, we use the NorthEuraLex dataset, a curated list of cross-linguistically common inanimate nouns (Dellert et al., 2020). Table 1 shows the counts for the remaining tokens for all analyzed languages for which we retrieved word representations. Next, we describe the procedure for computing the non-contextual word representations.

The model described in §3 relies on a representation of nominal lexical semantics—specifically, a representation independent (in the probabilistic sense) of the distributional properties of the noun.⁶

We train word2vec (Mikolov et al., 2013) on modified Wikipedia corpora. As found in Omrani Sabbaghi and Caliskan (2022), word representations learn the association between a noun and its grammatical gender in grammatically gendered languages. Thus, we first lemmatize the corpus with Stanza as discussed above. This step should remove any spurious correlations between a noun’s morphology and its meaning. Second, we remove all adjectives from the corpora. Because our goal is to predict the distribution over adjectives from a noun’s lexical semantic representation, that distribution should not, itself, be encoded in the semantic representation. We construct representations of length 200 through the continuous skip-gram model with negative sampling with 10 samples using the implementation from gensim.⁷ We train these non-contextual word representations on the Wikipedia data described above. We ignore all words with a frequency below five and use a symmetric context window size of five.

In addition to those representations derived from word2vec, we also derive lexical representations using WordNet (Miller, 1994). Because WordNet is a lexical database that groups words into sets of synonyms (synsets) and links synsets together by their conceptual, semantic, and lexical relations, representations of meaning based on WordNet are unaffected by biases that might be encoded in a training corpus of natural language. Following Saedi et al. (2018), we create word representations by constructing an adjacency matrix of WordNet’s semantic relations (e.g., hypernymy, meronymy) between words and compressing this matrix to have a dimensionality of 200 for each of the languages in this study: German, Hebrew, Spanish, Polish, and Portuguese (Siegel and Bond, 2021; Ordan and Wintner, 2007; Gonzalez-Agirre and Rigau, 2013; Piasecki et al., 2009; de Paiva and Rademaker, 2012). We access and process these WordNets using the Open Multilingual WordNet (Bond and Paik, 2012). We report statistics on these WordNets in Table 2.

We now discuss how we validate our lexical representations. Because we construct the word2vec representations using modified corpora, it is reasonable to fear that those modifications would hinder the representations’ ability to encode an adequate approximation to nominal lexical semantics. Thus, for each language, we evaluate the quality of the learned representations by calculating the Spearman correlation coefficient of the cosine similarity between representations and the human-annotated similarity scores of word pairs in the SimLex family of datasets (Hill et al., 2015; Leviant and Reichart, 2015; Vulić et al., 2020). A higher correlation indicates a better representation of semantic similarity. We report the Spearman correlation of the representations for each language in Table 3. Especially for representations generated using WordNet for languages with sparsely populated WordNets (see Table 2), the representational power is relatively low (as measured by the Spearman correlation), which may influence conclusions of downstream results for these languages. We note that if the representations are very bad—such that gender is completely unpredictable from the noun meaning representation and p_G(g_n∣n) = p_G(g_n)—then MI(A;G) =MI_do(A;G) because the edge in the graphical model from N to G is effectively removed.

We now outline the methodology of our study, starting with parameter estimation of the model in §5.1 and plug-in estimation of MI(A;G) in §5.2. We conduct two experiments: First, for a point of comparison, we replicate Williams et al.’s (2021) study to estimate MI(A;G) for each of the five languages (§5.3); and second, we produce a causal analog of Williams et al. (2021) (§5.4). Using the notation of §3, in the second experiment, we estimate MI_do(A;G) for each of the five languages. Finally, in §5.5, we discuss our permutation testing methodology.

To do this, we train a model from scratch using 5-fold cross-validation on a subset of 100 adjectives to estimate p_A(a∣n, g) with a random permutation of the gender labels and use that model to compute the pair-wise mutual information estimates between adjective distributions on the test set as described earlier for k = 2,000 times (a total of 10,000 runs). We design and run a permutation test to determine whether the mutual information between the adjective distributions conditioned on different genders is equal to the mutual information between the adjective distributions from a model trained on perturbed gender labels. We determine the significance of our result by evaluating the proportion of times that MI_do(A;G), as computed using the non-permuted training set, is lower than one computed using randomly permuted genders during training. We choose the standard significance level of α = .05; that is, when we observe a p-value lower than .05, we reject the null hypothesis, which posits no difference in mutual information between models trained on original and perturbed gender labels.

First, we validate our model by comparing the model-based estimation of MI(A;G) to the method presented in Williams et al. (2021), the plug-in estimation of MI(A;G). Then, we employ our causal graphical model to investigate whether there is evidence for the neo-Whorfian claim that the grammatical gender of a noun influences the adjective chosen to modify this noun, even when we control for the meaning of the noun.

We first validate our model by comparing its results to Williams et al.’s (2021) plug-in estimate of MI(A;G). If the results of both of these estimates are comparable, we have evidence that our model indeed captures the relation between grammatical gender and adjective choice. We present the results in Figure 2. We observe a substantial relationship between grammatical gender and adjective usage based on the plug-in and model-based MI(A;G) estimates replicating the results of Williams et al. (2021). The estimates of the model-based MI(A;G) computed using both word2vec and WordNet representations, and the plug-in MI(A;G) lie between 0.2 and 0.5, with the estimates of the model-based approach being consistently higher (with the exception of German) than the estimates of the plug-in MI(A;G). Thus, the non-zero estimates of the model-based MI(A;G) indicate that some relationship exists between a noun’s grammatical gender and adjective usage.

Given the above result, we are interested in whether the strength of this relationship is mitigated when controlling for the meaning of a noun. We present the estimates of the model-based MI_do(A;G) in Table 4 and compare them to the model-based estimates of the MI(A;G). While we observe evidence for the influence of grammatical gender on adjective choice in a non-causal setup based on MI(A;G), this relationship shrinks to close to 0 when we control for noun meaning in our causal model trained using both word2vec and WordNet representations.

For completeness, we test for the presence of a difference between the size of the MI_do(A;G) of our model and a model trained on randomly perturbed gender labels based on a subset of adjectives. We reject the null hypothesis that the distributions are exactly the same for all languages and representations’ settings.

We find that the interaction between the grammatical gender of inanimate nouns and the adjectives used to describe those nouns all but disappears when controlling for the meaning of those nouns, for all five analyzed gendered languages. While the order of magnitude of MI_do(A;G) measured with our model is significantly different from that of a model trained on random gender labels, it remains minuscule in absolute terms. This minor difference points towards the absence of a meaningful causal relationship between a noun’s gender and its modifiers in the languages studied. Thus, we provide an additional piece of evidence against the neo-Whorfian hypothesis in noun–adjective patterns.

Although the size of the overall effect is small, it is possible that the effect of gender on adjective choice is stronger for some words than others. Future work could explore whether there is evidence of a noticeable effect of gender on adjective choice for a more restricted set of inanimate nouns, e.g., referring to artifacts or body parts. Such evidence could perhaps support a weakened version of the neo-Whorfian hypothesis.

Our results hold for both of the word representation conditions, word2vec, and WordNet. Notably, in comparing the two, we find that using WordNet representations consistently results in a lower MI_do(A;G) than word2vec for all languages analyzed in this study. One possible explanation for this difference is that, despite our efforts to make non-contextual word2vec representations, these word2vec representations may still encode some signal regarding gender from the remaining context (such as verb choice or adjacent gendered pronouns in the corpora). If these word2vec representations contain unwanted context-based gender information in addition to the noun meaning, it could result in overestimating MI_do(A;G). Furthermore, since WordNet representations are created independently from any context within a corpus, they should not contain any signal related to grammatical gender, which may therefore be reflected in the consistently lower MI_do(A;G).

We note several choices in the experimental setup which may influence this analysis. First, while we experiment with NorthEuraLex, which furnishes us with a list of inanimate nouns, the dataset excludes rarer nouns for which an effect might be observed.⁹ Second, while non-contextual word representations are the current de facto proxy for lexical semantics, they remain a proxy and are fundamentally limited. Furthermore, in our effort to estimate word2vec representations for noun meaning without encoding gender-based context, we chose to remove some words in the context but not others. Specifically, while we remove adjectives which may carry signals of gender from the training corpora, we do not remove other parts of speech (e.g., verbs) under the reasoning that removing them may damage the training corpora too much for word2vec to effectively learn noun meanings.¹⁰ Future work can also explore improved representation methods for noun meaning. For example, Recski et al. (2016) find that creating non-contextual word representations using a combination of word2vec, WordNet, and concept dictionaries can yield a better representation of meaning (i.e., achieving state-of-the-art correlation with the human-annotated similarity scores). Third, the corpus choice (and subsequently the noun–adjective pairs on which we conduct our analysis) may factor into the results. It is possible that when applied to other corpora (e.g., more colloquial ones like Reddit), this method may yield different results. Fourth, the choice of languages analyzed further limits this study to languages with up to three gender classes. Future work can investigate languages with more complex gender systems. Finally, our modeling approach assumes that the gender of a noun is influenced solely by its meaning. However, prior work has indicated that there are other factors that influence the grammatical gender of nouns such as their phonology and morphology (Corbett, 1991). Therefore, future work should investigate more complex graphical models in order to account for other confounding factors.

In this paper, we introduce a causal graphical model that jointly models the interactions between a noun’s grammatical gender, its meaning, and adjective choice. We employ our model on five languages that exhibit grammatical gender to investigate the influence of nouns’ gender on the adjectives chosen to describe those nouns. Replicating the findings of Williams et al. (2021), we find a substantial correlation between grammatical gender and adjective choice. However, taking advantage of our causal perspective, we show that when controlling for a noun’s meaning, the effect of gender on adjective choice is marginal. Thus, we provide further evidence against the neo-Whorfian hypothesis.

The code and data necessary to replicate our empirical findings may be found at https://github.com/rycolab/neo-whorf.

This work is mostly funded by Independent Research Fund Denmark under grant agreement number 9130-00092B. Thanks to Nadav Borenstein for proofreading an earlier manuscript draft.