Semantic Drift in Multilingual Representations

Aligning the meaning of multiple languages in a joint representation to overcome language barriers has challenged humankind for centuries. Multilingual analyses range from the first known parallel texts on the Rosetta Stone through centuries of lexicographic work on dictionaries to online collaborative resources like Wiktionary (Meyer and Gurevych 2012) and Babelnet (Navigli and Ponzetto 2010). These resources vary in their semantic representations, but they rely mostly on symbolic approaches such as glosses, relations, and examples. In the last decade, it has become a common standard in natural language processing to take a distributional perspective and represent words, phrases, and sentences as vectors in high-dimensional semantic space. These vectors are learned based on co-occurrence patterns in corpora, with the objective that similar words should be represented by neighboring vectors. For example, we expect table and desk to appear close to each other in the vector space.

Recently, approaches to unifying these monolingual semantic representations into a joint multilingual semantic space have become very successful (Klementiev, Titov, and Bhattarai 2012; Vulić and Korhonen 2016; Conneau et al. 2017). The goal is to assign similar vectors to words that are translations of each other without affecting the monolingual semantic relations between words. For example, table should appear close to its Italian translation tavola without losing the proximity to desk which should in turn be close to the Italian scrittoio.

Cognitively inspired analyses have shown that the semantic organization of concepts varies between languages and that this variation correlates with cultural and geographic distances between language families (Eger, Hoenen, and Mehler 2016; Thompson, Roberts, and Lupyan 2018). We define this phenomenon as multilingual semantic drift and analyze to what extent it is captured in multilingual distributional representations. To this end, we propose a methodology for quantifying it that is based on the neuroscientific method of representational similarity analysis. Our approach uses a selected set of concepts and estimates how monolingual semantic similarity between concepts correlates across languages. We find that the results from our data-driven semantic method can be used to reconstruct language trees that are comparable to those informed by etymological research. We perform experiments on word-based and sentence-based multilingual models and provide both quantitative results and qualitative examples.

The article first introduces the most common architectures for multilingual distributional representations of words and sentences and then discusses approaches for quantifying the semantic structure that emerges in these models. These computational methods can be used to determine phylogenetic relations between languages. We elaborate on the data, the models, and the details of our analysis methods in an extensive methodology section. In a pilot experiment, we first evaluate the translation quality of the models for our data sets. The remaining sections discuss the results for the representational similarity analysis, the language clustering, and the identification of semantic drift. The code is available at https://github.com/beinborn/SemanticDrift.

The large success of monolingual distributional representations of words gave rise to the development of representations for longer sequences such as phrases and sentences. Researchers soon moved from monolingual to multilingual space and developed methods to obtain comparable representations for multiple languages. In this section, we introduce the related work for creating multilingual representations for words and sentences, and discuss approaches for capturing semantic structure and phylogenetic relations.

Approaches for constructing multilingual representations for words can be distinguished into two main classes: mapping models and joint models (Ruder, Vulić, and Søgaard 2019).¹ The multilingual modeling techniques are very similar to those applied on learning shared representations for multiple modalities—for example, vision and language (Beinborn, Botschen, and Gurevych 2018; Baltrušaitis, Ahuja, and Morency 2019).

Mapping approaches are based on pre-trained monolingual representations of two languages (the source and the target language) and aim to project the representations from the semantic space of the source language to the target space. This approach is based on the idea that the intralingual semantic relations are similar across languages (Mikolov, Le, and Sutskever 2013) and can be exploited to learn a linear projection from one language to the other. The linear projection is learned based on a bilingual seed dictionary that provides a link between the semantic spaces (Gouws and Anders 2015b; Vulić and Korhonen 2016) or by aligning information from parallel corpora. In general, mapping models are directional and map representations from one language to the other. Faruqui and Dyer (2014) propose to instead map both representations into a joint space by applying canonical correlation analysis. During training, they enforce maximum correlation of representations for words that are known to be translations of each other.

For joint approaches, both representations are learned simultaneously by using parallel corpora for training. These models jointly optimize the objectives for monolingual and crosslingual similarity. The monolingual objective is based on co-occurrence patterns observed in context and is similar to those that are commonly applied for training monolingual representations—for example, the skip-gram objective in word2vec (Mikolov et al. 2013) or variants of it (Luong, Pham, and Manning 2015). The crosslingual objective can be derived from word alignments (Klementiev, Titov, and Bhattarai 2012), sentence alignments (Gouws, Bengio, and Corrado 2015), or document alignments (Fung and Yee 1998; Søgaard 2016).

Most of the described models are inherently bilingual rather than multilingual. Duong et al. (2017) and Levy, Søgaard, and Goldberg (2017) show that learning representations for multiple languages simultaneously is beneficial because it facilitates transfer learning between closely related languages. We refer the interested reader to the detailed survey by Ruder, Vulić, and Søgaard (2019) for further explanations on the mathematical foundations of crosslingual representations.

The quality of a multilingual model is dependent on the quality of the crosslingual signal. Several approaches are aimed at enriching the signal by incorporating additional resources, such as visual cues (Bergsma and Van Durme 2011; Vulić et al. 2016) or syntactic information (Duong et al. 2015). Unfortunately, aligned multilingual corpora are usually scarce in low-resource languages. For covering a wider range of languages, self-supervised approaches that do not rely on predefined alignments have been developed.

Smith et al. (2017) and Hauer, Nicolai, and Kondrak (2017) derive the crosslingual information by simply exploiting identical character strings from loanwords or cognates. As this only works for languages with the same alphabet, Artetxe, Labaka, and Agirre (2017) go one step further and instantiate their model only with aligned digits. Conneau et al. (2017) and Zhang et al. (2017) do not use any parallel data and apply adversarial training to optimize the mapping between languages. Their generator tries to map the source words into the target space, while the discriminator attempts to distinguish between the target representations and the mapped source representations. As both the discriminator and the generator get better at their task, the mapped representations resemble more the target representations. More recently, their approach has been transformed into a generative model using variational autoencoders (Dou, Zhou, and Huang 2018).

In this work, we use the Muse model, which has become popular through its performance in self-supervised settings (Conneau et al. 2017). The model is based on monolingual representations (fasttext) that are trained on merged text data from Wikipedia and the CommonCrawl corpus and obtain good results for a wide range of languages (Bojanowski et al. 2017). Whereas the Wikipedia data alone might contain a small domain bias because the articles cover varying ranges of topics across languages, the immense size of the CommonCrawl corpus provides a good approximation of actual written language use. It contains 24 terabytes of raw text data crawled from the Web summing up to several billions of tokens for each language (Grave et al. 2018). In order to ensure high quality for our experiments, we rely on multilingual representations obtained in a supervised fashion using a ground-truth bilingual dictionary.² The entries of the dictionary serve as anchor points to learn a mapping from the source to the target space that is optimized by Procrustes alignment (Schönemann 1966).

The need for developing multilingual representations for sentences is most prevalent in the field of machine translation. Already in the 1950s, the idea of an interlingua that could serve as a bridge between multiple languages emerged (Gode and Blair 1951). The idea was further pursued by searching for a formalism that should represent the semantic content of a sentence independent of the language in which it is realized (Richens 1958). Similar ideas have driven the development of logic formalisms such as Montague grammars (Montague 1970). With the incredible success of powerful neural networks, it has currently become widely accepted that the most suitable form for such interlingual or multilingual representations are high-dimensional vectors.

Although discussing the wide range of machine translation literature is beyond the scope of this article, we briefly describe two main state-of-the-art models: encoder-decoder architectures and the transformer architecture.

Machine translation is commonly interpreted as a sequence-to-sequence learning problem. Sutskever, Vinyals, and Le (2014) paved the way for fast developments on so-called encoder-decoder architectures. The encoder reads the input and learns to transform it into an intermediate representation that is then fed to the decoder to generate the translation of the sentence in a target language. Both the encoder and the decoder can be realized as different types of recurrent neural networks and can be combined with different techniques of attention (Bahdanau, Cho, and Bengio 2015). Recently, bidirectional long short-term memory networks (BiLSTMs) have proven to be a good choice for modelling language (Peters et al. 2018). Schwenk (2018) show that using a joint BiLSTM encoder for all input languages combined with max-pooling over the last layer yields more robust sentence representations. After training, the decoder that is responsible for the translation generation is discarded and the output of the trained encoder is used as universal sentence representation. These sentence representations can be interpreted as “sort of a continuous space interlingua” (Schwenk and Douze 2017, p.158). We use a pre-trained version of this model that is called Laser (Artetxe and Schwenk 2019).

More recently, Vaswani et al. (2017) introduced the transformer model as a more sophisticated architecture for sequence to sequence transduction. Its underlying architecture follows the encoder-decoder paradigm, but no recurrent connections between tokens are used, which reduces the training time for the model. In order to capture relations between tokens, a complex attention mechanism called multi-headed self-attention is applied and combined with positional encoding for signaling the order of tokens. Because of its success, variants of the transformer model for machine translation are currently being developed in a very fast pace. In the past, language modelling has commonly been interpreted as a left-to-right task, similar to incremental human language processing (Rosenfeld 2000). As a consequence, the self-attention layer could only attend to previous tokens. Devlin et al. (2019) argue that this approach unnecessarily limits the expressivity of the sentence representation. They propose to change the training objective from predicting the next word to predicting a randomly masked token in the sentence by considering both the left and right context. This task is also known as the cloze task (Taylor 1953). Devlin et al. (2019) use this training objective to train a multilayer bidirectional transformer (called Bert) and find that it strongly outperforms the previous state of the art on the GLUE evaluation corpus (Wang et al. 2018). By now, they have also released a multilingual version of Bert for 104 languages.³

Bert and Laser obtain comparable results on the crosslingual entailment data set (Conneau et al. 2018). For this article, we decided to use Laser because the model already outputs sentence representations that have a uniform dimensionality independent of the length of the sentence. This makes it possible to avoid additional experimental parameters for scaling the dimensionality of the sentence representations. The model has been trained by combining multiple multilingual parallel corpora from the Opus Web site (Tiedemann 2012) accumulating to a total of 223 million parallel sentences (Artetxe and Schwenk 2019).⁴ Note that the sentence-based model is optimized for translation whereas the word-based model aims at optimizing both monolingual semantic similarity and crosslingual translation constraints. These different training objectives might have an influence on the model’s ability to capture semantic differences between languages.

Multilingual representations are commonly evaluated based on their performance on downstream tasks such as bilingual lexicon induction (Vulić and Korhonen 2016) and machine translation (Zou et al. 2013). More indirectly, multilingual representations are used for crosslingual transfer in tasks such as information retrieval, or document classification (Klementiev, Titov, and Bhattarai 2012). From a semantic perspective, multilingual representations are evaluated by comparing distances in the vector space with crosslingual semantic similarity judgments by humans (Cer et al. 2017). Sentence representations are often tested by their ability to distinguish entailment relations between sentences (Conneau et al. 2018). Most of these evaluations are simply multilingual extensions of monolingual evaluation tasks. These tasks ignore an important aspect of multilingual representations, namely, the relations between languages.

Languages are fluid cultural constructs for communication that undergo continuous finegrained structural transformation due to emergent changes in their usage. A large body of work in historical linguistics aims to quantify how languages evolve over time and how different languages are related to each other. For example, Italian and Spanish both evolved from Latin and are thus more similar to each other than to Eastern European languages like Polish. One way of visualizing the typological relations between languages are phylogenetic trees. As a complement to historical research, computational analysis methods aim to automatically reconstruct phylogenetic relations between languages based on measurable linguistic patterns. We briefly introduce three main approaches for measuring language similarity:

• (1) 

  Lexical overlap: Earlier work on reconstructing phylogenetic relations mostly relies on determining lexical overlap between languages based on manually assembled lists of cognates (Nouri and Yangarber 2016), which is a cumbersome and subjective procedure (Geisler and List 2010). Several methods for automatically extracting cognates exist (e.g., Serva and Petroni 2008), but these approaches rely on the surface structure of a word. Beinborn, Zesch, and Gurevych (2013) use character-based machine translation to identify cognates based on regular production processes, but their method still cannot capture the cognateness between the English father and the Italian padre, for example. For the methodology that we propose here, we abstract from the surface appearance of the word and focus on its semantic properties. As a consequence, we do not need to transliterate words from languages with different alphabets.

• (2) 

  Structural similarity: The similarity between languages is often measured by the similarity of their structural properties (Cysouw 2013). The World Atlas of Language Structures (wals) lists a large inventory of structural properties of languages including phonological, grammatical, and lexical features.⁵ Rabinovich, Ordan, and Wintner (2017) analyze translation choices and find that the syntactic structure of the source language is reflected in English translations. Recently, Bjerva et al. (2019) build on their work and analyzed the structural similarity between languages using phrase structure trees and dependency relations. Both approaches are able to reconstruct a phylogenetic tree solely based on syntactic features of the translation. We apply the same evaluation method for estimating the quality of the generated tree in Section 6.3, but we estimate the similarity of languages based on their semantic organization.

• (3) 

  Semantic organization: Recent works indicate that semantic similarity between languages can also serve as a proxy for determining language families. Eger, Hoenen, and Mehler (2016) find that semantic similarity between languages correlates with the geographic distance between countries in which the languages are spoken. In a similar vein, Thompson, Roberts, and Lupyan (2018) find that semantic similarity between languages is proportional to their cultural distance. In these works, semantic structure is approximated by graph-based networks that indicate the associative strength between words. The challenge lies in aligning these structures across languages and accounting for complex semantic phenomena such as polysemy and context sensitivity (Youn et al. 2016).

Semantic drift is mostly known from diachronic studies where it indicates the change of meaning over time (Li et al. 2019; Frermann and Lapata 2016; Hamilton, Leskovec, and Jurafsky 2016b).⁶ Popular examples are the meaning drift of gay from cheerful to homosexual over the years, or the transformation of cognates into false friends as in gift which today means poison in German (but originally referred to something given). Recently, a wide range of distributional approaches have been developed for measuring diachronic semantic change (see surveys by Tahmasebi, Borin, and Jatowt [2018] and Kutuzov et al. [2018] for an overview).

Semantic drift can also be observed across languages because even an exact translation of a word or a phrase does not share all semantic associations. For example, pupil could be translated to Spanish as pupila, but the Spanish phrase would only be associated with the eye and not with school children. These differences in the semantic scope of a word can lead to important differences in translation. Conneau et al. (2018) observe that the English term upright had been translated to Chinese as sitting upright. As a consequence, the original sentence entailed standing in their multilingual entailment corpus, but the translation violated this entailment relation. In this work, we analyze to which extent multilingual models preserve these semantic drifts. Faruqui and Dyer (2014) claim that multilingual projection can contribute to word sense disambiguation. For example, the polysemous English word table is translated to tafel in Dutch if it refers to a kitchen table, and to tabel if it refers to a calculation matrix. They provide a qualitative example for the word beautiful to show that synonyms (pretty, charming) and antonyms (ugly, awful) are better separated in multilingual spaces. Dinu, Lazaridou, and Baroni (2014) analyze zero-shot learning in multilingual and multimodal models and conversely find that fine-grained semantic properties tend to be washed out in joint semantic space. They describe the “hubness problem” as the phenomenon that a few words (the hubs) occur among the nearest neighbors for a large number of other words and show that this problem is more severe in mapped representational spaces.⁷

The approaches for comparing semantic structure over time and over languages are similar. Concepts are represented as vectors in high-dimensional semantic space and can be compared diachronically by calculating different vectors for each time epoch using historical corpora (Hamilton, Leskovec, and Jurafsky 2016b; Rosenfeld and Erk 2018) or multilingually by calculating different vectors for each language (Asgari and Mofrad 2016). As the global position of a vector is often not comparable across corpora, the semantic drift is approximated by determining changes in the relative position of a concept within its local neighborhood (Hamilton, Leskovec, and Jurafsky 2016a). The resulting semantic networks can be compared by representing them as graph structures (Eger, Hoenen, and Mehler 2016; Youn et al. 2016) or second-order similarity matrices (Hamilton, Leskovec, and Jurafsky 2016a; Thompson, Roberts, and Lupyan 2018). The distance between these computational structures can then be used as an indicator for the amount of drift. An additional challenge for multilingual comparisons lies in determining the alignment of concepts across languages. Our computational approach is most similar to the method by Thompson, Roberts, and Lupyan (2018). They use a much larger set of stimuli (more than 1,000) for which gold translations are available and analyze their representations in monolingual embedding models. In our work, we focus on multilingual representations and analyze the crosslingual similarity that emerges from the model. We extract translations in a data-driven way by taking the nearest semantic neighbor in semantic space instead of relying on translation resources. The details of our methodology are described in the following section.

In this section, we detail the methodology applied for our experiments. We provide information on the data, the multilingual models, and the methods used for comparing representational spaces.

We perform our word-based experiments with a set of stimuli that have been selected to be universal representatives of the most important semantic concepts. For the sentence-based experiments, we extract sentences from a parallel corpus. More information on the data and the languages can be found in the Appendixes. As we are using very common and frequent data, it is likely that the stimuli have also occurred in the training data of the models. However, we are interested in examining the resulting representational relations between stimuli and the effect of the multilingual training regime of the models.

The American linguist Morris Swadesh composed several lists of so-called language universals: semantic concepts that are represented in all languages (Swadesh 1955). His lists have been revised multiple times and have also been subject to strong criticism (Geisler and List 2010; Starostin 2013). Nevertheless, they are still a popular tool in comparative linguistics and have been collected for a large range of languages and dialects. We are using the extended list of 205 different English words that is available on Wiktionary (Wiktionary Contributors 2019); see Appendix C.

Pereira et al. (2018) selected semantic concepts by performing spectral clustering on word representations obtained from Glove (Pennington, Socher, and Manning 2014). They selected concepts by maximizing the variation on each dimension of the semantic space. After pre-processing, they manually selected 180 words (128 nouns, 22 verbs, 23 adjectives, 6 adverbs, 1 function word), claiming that their selection best covers the semantic space (see Appendix D). The concepts were originally selected to serve as stimuli in brain imaging experiments on semantic language processing in humans. The Pereira list overlaps with the Swadesh list for 20 words. We ignore the word argumentatively because it is not in the vocabulary of the Muse model.

Koehn (2005) extracted the Europarl corpus from the proceedings of the European Parliament. It includes sentence-aligned versions in 21 European languages. As the corpus contains formal political language, short sentences are often captions or names of documents and long sentences tend to be overly complex. We thus decided to restrict our analysis to sentences of medium length ranging from 6 to 20 words. In order to better control for sentence length, we extract three sets of 200 random sentences each conforming to three length constraints. The set of short sentences consist of 6–10 words, mid sentences of 11–15 words, and long sentences of 16–20 words. We restrict the 21 languages to the 17 used in Rabinovich, Ordan, and Wintner (2017). Whereas they use only English sentences (which are translations) and examine the syntactic structure of the sentence with respect to the language of the source sentence, we use translations into multiple languages and keep the set of sentences constant.

We use two different freely available pre-trained multilingual representations for our experiments, which have been reported to achieve state-of-the-art performances. For our experiments with words, we use Muse representations (Conneau et al. 2017) and for our sentence-based experiments, we use Laser representations (Artetxe and Schwenk 2019). Their architectures are described in Section 2.

The Muse representations are available for 29 languages, aligned in a single vector space. For our experiments, we ignore Vietnamese because spot checks indicated quality issues. For all other languages (see Appendix A for a complete list), we load a vocabulary of 200,000 words. The model encodes every word as a 300-dimensional vector.

The Laser model generates a sentence representation as a list of tokens. Each token is assigned a vector representation that reflects the contexts in which it occurs. We are using the pre-trained multilingual model that is available for 93 languages. The model encodes every sentence as a 1,024-dimensional vector independent of the length of the sentence that facilitates the comparison across sentences.

Comparing the relations in representational spaces is an interdisciplinary problem that is routed in linear algebra and has applications in a large range of research areas. In natural language processing, most work on comparing monolingual representational spaces targets the goal of building better multilingual representations. Canonical correlation analysis (Faruqui and Dyer 2014; Ammar et al. 2016), Kullback-Leibler divergence (Dou, Zhou, and Huang 2018), and Procrustes alignment (Conneau et al. 2017) are only a few methods to maximize the similarity between two representational spaces. Recently, similar methods are being used to compare the hidden representations in different neural models (Raghu et al. 2017). In this article, we apply a method that has been introduced to compare representations obtained from computational models with neuroimaging data of human brain activations (Kriegeskorte, Mur, and Bandettini 2008). For this method, the representational relations are first evaluated for each modality individually in a representational similarity matrix using common similarity measures such as Euclidean, Cosine, or Mahalanobis. In a second step, the two matrices are compared with each other using Spearman correlation to analyze whether the identified relations are similar for the two representational modalities. Representational similarity analysis can also be used to compare different modalities of a computational model (Abnar et al. 2019). In our case, a modality refers to a language. In the following, we formally describe the method and introduce the terminology used for the remainder of the article. For simplicity, we focus on words as the unit of analysis, but the same methodology is used for analyzing sentences.

Note that the similarity vector is comparable to the similarity vector by Hamilton, Leskovec, and Jurafsky (2016a), which is used to measure semantic drift over time. In our case, the group of “neighbors” to analyze is set to the words in our list to ensure crosslingual comparability. The underlying concept is also comparable to semantic network analyses (Li et al. 2019; España-Bonet and van Genabith 2018).

Based on the second-order similarity matrix calculated over all languages, we can identify relations between languages. Similarly to Rabinovich, Ordan, and Wintner (2017), we perform hierarchical language clustering using Ward’s variance minimization algorithm (Ward Jr 1963) as a linkage method to attempt phylogenetic reconstruction. Ward’s method iteratively minimizes the total within-cluster variance by, at each step, using this objective as a criterion for selecting new pairs of clusters to merge. To measure the distance between data points, Euclidean distance is used. Whereas Rabinovich, Ordan, and Wintner (2017) use a large set of features as input, we only use the similarity value described earlier. This value captures to which extent semantic relations between words follow similar patterns in the two languages.

We apply our methodology for a series of experiments. We first estimate the quality of the multilingual models for our data sets and then present results for representational similarity analysis and language clustering.

The quality of monolingual word representations is commonly estimated by evaluating to what extent words that are semantically similar (such as lake and pond) can be found in close proximity to each other in the semantic space. For multilingual models, the goal is to minimize the representational distance between words that are translations of each other. We check the model quality for a stimulus by determining the nearest neighbor of the stimulus for each target language (see Equation (3)) and comparing it to a list of known translations.

The interpretation of the semantic space of sentence vectors is more complex because we can generate infinitely many possible sentences because of the compositionality of language. As a consequence, it is hard to define which sentences should be present in the neighborhood of a sentence even in monolingual space. A sentence with synonyms? The same sentence in another tense? The negated form of the sentence? When we are moving to multilingual space, the monolingual constraint remains fuzzy, but the multilingual training objective is clear: Sentences that are aligned as translations in parallel corpora should be close to each other.

The results for both word-based and sentence-based translation quality assessments are reported in Table 1 and discussed in more detail below (see Appendixes A and B for a list of languages and their abbreviations).

As the stimuli are presented without context, they might be translated with respect to any of their senses. In order to account for this wide range of translations, we use the multilingual resource Babelnet (Navigli and Ponzetto 2010). For each stimulus word, we collect the translations of all senses of all possible synsets and check whether any of these translations matches the nearest neighbor in the target language retrieved by the Muse model. As the words in the model are not lemmatized, we additionally check for close matches using the difflib-module in Python.¹¹ We also count a match if the nearest neighbor of the word is the word itself used as a loanword in the target language. We noticed that the coverage of Babelnet is insufficient for the Swadesh stimuli because it does not contain simple words like you or with. To account for this, we additionally consult ground-truth dictionaries.¹²

The translation quality for the Muse model is not perfect, but higher than reported in Artetxe and Schwenk (2019) because we use a smaller set of stimuli. The quality is better for the Pereira stimuli (row 1) than for the highly frequent Swadesh stimuli (row 2). As frequency correlates with polysemy, the translation options for these stimuli might be more fuzzy. We had a closer look at the incorrect entries for some languages and noted that the nearest neighbor usually points to semantically highly related words. For example, the nearest German neighbor of the stimulus silly is lächerlich, which means ridiculous in English. We observe that the translation quality tends to decrease for languages that are less similar to English. This indicates that the model provides an interesting testbed for examining semantic drift movements across languages.

Given that the word translations are an integral part of our methodology, we test the influence of the translation quality on our semantic drift estimations in Section 6.4. We extract reference translations from the NorthEuraLex database (Dellert and Jäger 2017) to replace the nearest neighbor method for the crosslingual representational similarity analysis.

For the sentence-based experiments, we count a perfect match if the nearest neighbor of a sentence in the target space matches the translation of the sentence in the corpus. Schwenk and Douze (2017) refer to this method as similarity search. We find that the quality is almost flawless independent of the sentence length. The results indicate that semantic drift phenomena are more likely to occur in the word-based model because the sentence-based model exhibits less variation across languages. It is optimized with respect to multilingual translation, whereas the word-based model balances monolingual semantic similarity and crosslingual translation constraints and optimizes them jointly.

In this section, we illustrate the steps of our analysis method. We first look at intralingual semantic relations and then perform a crosslingual representational similarity analysis. We focus on the word-based model because example words can be visualized more intuitively than full sentences.

We first extract the English vectors for all words in our list. We analyze the cosine similarity between the vectors and construct a representational similarity matrix as described in Equation (1). We then extract the translations for each word in our lists as described in Equation (3) to construct representational similarity matrices for all languages.

Figure 1 illustrates example matrices for a subset of the five words small, short, child, wife, mother for English, Spanish, and Russian. It can be seen that the similarity patterns are comparable, but we also note some differences. For example, the extracted Spanish words niño and pequeño are more similar to each other than their translations child and small. We assume that this is due to the fact that both small and little are translated as pequeño in Spanish. This illustration indicates that semantic relations vary slightly across languages. Note that the nearest Russian neighbor is not the most intuitive translation for small because it is a plural form (cosine similarity: 0.67). This effect occurs because the vocabulary in the Muse model is not lemmatized. We observe that the cosine similarity (0.67) is quite similar for alternatives like (0.65) and (0.61). This indicates that it might be reasonable to analyze the top n nearest neighbors and/or only work with lemmas to obtain purer results from a linguistic perspective.

The intralingual similarity matrices described earlier serve as the basis for the crosslingual representational similarity analysis. We measure the correlation of the semantic similarity for each pair of languages as described in Equation (5). The resulting matrix is illustrated in Figure 2 for five selected languages. It can be seen that languages like Spanish (es), Portuguese (pt), and French (fr) have highly similar semantic patterns. For German (de), the similarity is slightly lower, and Finnish (fi) stands out as not being very similar to the other languages.

The second-order similarity matrix only indicates the average correlation between the similarity vectors of the words. For linguistic analyses, it is more interesting to look at the behavior of individual words and word pairs. In Figure 3, we plot the word pairs with the highest variance in similarity across languages. It is interesting to see that all six words are adjectives. Word representations are mostly analyzed on nouns (Finkelstein et al. 2002) and sometimes on verbs (Gerz et al. 2016). Faruqui and Dyer (2014) discuss that separating synonyms and antonyms in word representations can be tricky because they tend to occur in very similar contexts. We find that the nearest French neighbor of both left and right is gauche meaning left. The same phenomenon occurs for Catalan. For Slovenian, both words are translated to desno meaning right. For the pairs big-great and poor-bad, we observe that they are translated to the same word in some languages, which is not surprising as they are likely to occur in similar contexts. However, the nearest neighbor of big is big in many languages because it is often used as a loanword. Unfortunately, the loan word exhibits different semantic properties because it is only used in specific contexts (e.g., products or other named entities), whereas a translation would be used in more common contexts. This explains the low similarity between big and great for many languages. Our findings indicate that the methodology we introduce for analyzing crosslingual relations can also be used to identify flaws of the computational model.

In these three examples, it seems as if the cosine similarities are generally higher for Estonian, Hungarian, Greek, and Hebrew, whereas they are constantly low for Italian, Portuguese, and Spanish. In order to verify whether this observation points to a systematic pattern, we checked the mean and variance scores for the pairwise similarities, but the scores were comparable for all seven languages.

We use the result of the representational analysis to run a hierarchical clustering algorithm over the languages. The clustering is only based on the semantic similarity scores for pairs of languages (see Section 3 for details). We first discuss the word-based and the sentence-based results separately and then perform a quantitative evaluation.

Surprisingly, our computationally generated trees in Figure 4 resemble the trees that are commonly accepted by linguistic experts quite closely. We cross-check our observations against the renowned linguistic resource Glottolog (Hammarström et al. 2018) and observe a clear distinction between Western and Eastern European languages in the generated tree. It is even possible to identify a distinction between Germanic and Latin languages (with the exception of English). Obviously, the extracted cluster tree is not perfect, though. For example, Indonesian (id), Hebrew (he), and Turkish (tr) do not fit well with the rest and Romanian (ro) would be expected to be a little closer to Italian.

The subtree containing the languages Russian (ru), Ukranian (uk), Czech (cs), and Polish (pl) is grouped with other Slavic languages for the Swadesh tree and with the Germanic languages in the Pereira tree. This might be an artifact of our quite diverse set of languages spanning many language families including non-Indo-European ones like Finnish (fi), Hebrew (el), and Indonesian (id). Czech and German, for example, are quite related and share many cognates because of historical reasons, so that their closeness in the tree is explainable. The tree using the combined stimuli is more similar to the Swadesh version (see Appendix F, Figure F4).

Furthermore, it is interesting to note that similarly to the tree by Rabinovich, Ordan, and Wintner (2017), Romanian (ro) and Bulgarian (bg) are clustered together in our trees although they represent different language families (Romance and Slavic languages). Our observations indicate that language contact might be more relevant for semantic drift phenomena than a common ancestor language. The same argument could explain the vicinity of English (en) and French (fr). Our findings support the results by Eger, Hoenen, and Mehler (2016) and Thompson, Roberts, and Lupyan (2018) that showed that semantic similarity between languages correlates with their cultural and geographic proximity.

Figure 5b shows the results of the clustering using the sentence-based model. We see that the separation of Eastern and Western European languages works quite well, but the more finegrained relations between languages are less accurately clustered than for the word-based experiments. In particular, Dutch (nl) and Italian (it) should be noted as outliers. From a quantitative perspective, we find that the distances between languages (visualized on the y-axis) are much lower than for the word-based experiments.¹³ Recall that the Laser architecture is optimized for translating between multiple languages. Based on this training objective, it is plausible that subtle differences between languages tend to be smoothed out. In contrast, the word-based Muse model has explicitly been optimized to fulfill both the monolingual objective (preserve intralingual semantic relations) and the crosslingual objective (representational similarity of translations).

In order to better judge the quality of the language tree, we additionally perform quantitative experiments and compare the distance to a gold tree. For the 17 languages in the sentence-based model, we use the same tree as Rabinovich, Ordan, and Wintner (2017) that was developed by Serva and Petroni (2008) (see Figure 5a). For the word-based model, we reduce the set of languages to 20 Indo-European ones and adjust the gold tree accordingly (see Figure 6a). We calculate the distance between our trees based on the representational similarity analysis (see Figures 5b and 6b) to the gold trees as described in Equation (6).

As the distance score depends on the number of leaves, the results are not directly comparable for word-based and sentence-based experiments. We thus calculate the average distance score for a randomly generated tree over 50,000 iterations and report the change in quality of our model with respect to that baseline in Table 2. For this random baseline, instead of calculating the similarity matrix of languages as described in Section 3.3, we generate it randomly from a uniform distribution between 0.3 and 0.8 (we omit the extreme values because they are unlikely to be observed experimentally).¹⁴ This might not be the most plausible distribution, so we calculated an additional permutation baseline. We randomly permute the values of the similarity matrix by our model and average the results over 50,000 iterations.¹⁵

We see that the quality of our generated trees is considerably better than chance. In particular, the results for the Swadesh stimuli and the mid sentences stand out as strong improvements over a random baseline. The clustering results seem to negatively correlate with the translation quality of the model (see Table 1): Clustering works better for the word-based model than for the sentence-based model and better for Swadesh than for Pereira stimuli. We speculate that lower translation quality is a data-driven indicator of semantic distance. As a consequence, the differences between languages become more pronounced in our analysis, which leads to better clustering. These findings support our assumption stated in the previous section that models that are optimized for learning universal sentence representations smooth out the differences between language families. From an engineering perspective, this is a reasonable goal, but it might come at a cost of expressivity when it comes to more finegrained linguistic and cultural subtleties.¹⁶

It is interesting to see that the results for the permutation baseline are even worse than for the random baseline in the word-based setting. This shows that our methodology does pick up on the differences between languages. If this inductive bias is scrambled, the clustering results get worse than when treating all languages uniformly. For the sentence-based experiments, we do not observe this effect because the similarity scores are more homogeneous across languages.¹⁷

The results by Rabinovich, Ordan, and Wintner (2017) are slightly better than our word-based ones because they used hundreds of structural features whereas we only use a single semantic measure. In addition, they tackled a slightly different task as they only worked with English sentences, which are translations. It should be noted, that improvements for the sentence-based experiments are easier to obtain because the lower number of languages (17 vs. 20) leads to a lower number of possible combinations in the tree.

We have seen in Table 1 that the translations obtained by selecting the nearest neighbor for the word-based model are not always accurate. In this experiment, we test the influence of the translation quality on the semantic drift estimation. For comparison, we obtain reference translations from the NorthEuraLex database, which contains 1,016 distinct concepts (Dellert and Jäger 2017). Unfortunately, many concepts of our stimuli lists cannot be found in the database. We select concepts according to the following constraints instead:

If several concepts are mapped to the same word in English (for example fly refers both to the insect and to the movement), we only keep the noun concept because words are not sense-disambiguated in Muse. We only select concepts that are translated to single words in all our 20 reference languages and ignore, for example, cheap, which is translated to bon marché in French. In addition, we ignore number concepts such as four. All translations need to be found in the 200,000 loaded words from Muse, so we ignore infrequent concepts like hoarfrost. The selection process can be reproduced using our code on GitHub and yields 417 concepts and their translations for testing (see Appendix E for a list of the concepts).

We use these extracted reference translations to replace the nearest neighbor method and then perform crosslingual representational similarity analysis as before. The quality of the phylogenetic tree obtained by clustering over the crosslingual similarity matrix improves by +58.2% compared with the random baseline. When we use the nearest neighbor method with the same set of stimuli, the improvement is lower (+46.0%), but still satisfactory. These results indicate that using reference translations leads to more accurate results and that the choice of stimuli plays an important role (the quality for the NorthEuraLex stimuli is in between the results for the Swadesh and the Pereira stimuli; compare Table 2). The nearest neighbor method is a viable alternative if reference translations are not available or if the research question focuses on representational properties of the multilingual model. The choice of the multilingual training objective seems to have a stronger impact on the ability of the model to represent crosslingual semantic differences than the translation quality of the stimuli.

The clustering results indicate that the distances between the words vary in the semantic space of different language families. For a qualitative exploration, we have a closer look at semantic drift for three language pairs that are clustered closely together in our trees: Spanish and Portuguese (es, pt) for the Romance languages, German and Dutch (de, nl) for the Germanic languages, and Russian and Ukranian (ru, uk) for the Slavic languages. We include this analysis as an illustration of our methodology. The choice of the clusters and their size could be conveniently adjusted for any linguistic hypothesis.

For each word in the Pereira list, we calculate the semantic drift as described in Equation (8). In Figure 7, we visualize two examples with a high drift score for these clusters. The word representations have been reduced to two dimensions by applying principal components analysis on the joint representational space of all six languages (Wold, Esbensen, and Geladi 1987). For each language, we plot the Pereira words with the highest cosine similarity to lady and reaction. For readability, we always use the English word, but, for example, pain colored in red stands for the nearest Spanish neighbor of pain, which is dolor.

It can be seen that lady is close to religious for Portuguese and Spanish, but not for the other languages. We note, that the nearest neighbor for lady is not its translation, but the loanword itself (or its transliteration) for Dutch, German, and Russian. This explains the similarity to sexy and big, which are also used as loanwords in Dutch and German. The word reaction is a cognate originating from Latin in all six languages. The plot indicates that it is more closely associated with technical terms in the clusters (es, pt) and (de, nl), and with emotional terms in the cluster (ru, uk).

It should be noted that these examples only serve as anecdotal evidence and that the differences between the languages cannot always be observed when looking at only two dimensions. However, our methodology makes it possible to quantify semantic differences between words across languages. This can be used to better understand flaws of the computational representations (e.g., the observation that words tend to be represented by loanwords even when a more accurate translation exists), and the methodology can also generate hypotheses for historical linguistics when applied on a larger vocabulary. From an application perspective, analyses of semantic drift are particularly interesting for the field of foreign language learning. When understanding a foreign text, learners rely on background knowledge from languages they already know (Beinborn 2016). Phenomena of semantic drift can thus lead to severe misunderstandings and should receive increased attention in education.

We have introduced a methodology to analyze semantic drift in multilingual distributional representations. Our analyses show that by comparing the representational distances for a test set of about 200 words, we can reconstruct a phylogenetic tree that closely resembles those assumed by linguistic experts. These results indicate that multilingual distributional representations that are only trained on monolingual text and bilingual dictionaries preserve relations between languages without the need for any etymological information. Methods in lexicostatistics have previously been criticized for relying on subjective cognate judgments (Geisler and List 2010). A certain level of subjectivity might also be present in the “ground-truth” bilingual dictionaries used for the computational multilingual representations that were analyzed in this article. However, the large vocabulary should help to balance out potential biases.

So far, multilingual representations have mostly been evaluated based on their performance on specific tasks. In this article, we look beyond engineering goals and analyze the semantic relations between languages in computational representations. We find that the word-based model captures differences in the semantic structure that correspond to linguistic expectations. The sentence-based model, on the other hand, seems to be optimized to balance out subtle differences between language families. This might be a suitable scenario for obtaining better machine translation results, but for linguistic analyses the training objective would have to be adjusted toward maintaining some language diversity. Another important aspect is the training data of the computational models. The corpora used for the word-based model might be less balanced across languages and as a consequence, differences between languages are reinforced.

For future work, analyses of semantic drift in multilingual representations can serve two main purposes: From a technical perspective, they can indicate unwanted characteristics in the multilingual representations and steer processes for technical improvement of multilingual models. In our analyses, we have seen that words tend to be close to identical loanwords in the target space even if a more accurate translation is available. Loanwords often find their way into a language to add nuances to the semantic inventory. As a consequence, they tend to occur only in specific contexts that call for these nuances. The semantic relations to other words can thus be biased due to the introduction of the loanword. In addition, we find that adjectives are not well separated from their antonyms in the semantic space. This indicates that relying on co-occurrence patterns might not be sufficient for capturing semantic relations in word classes other than nouns.

From a linguistic perspective, our methods provide a quantitative means to study linguistic phenomena across languages. The development of multilingual computational models opens up new possibilities for comparative linguistics. In this article, we have laid out a methodology to query these models for semantic drift. The results of these queries can be used to generate hypotheses for historical linguistics and social linguistics because they indicate similarities in the organization of semantic concepts. For linguistically motivated analyses, it is worthwhile to pay close attention to the translation quality.

Our word-based experiments used English as the anchor language for obtaining translations. This is not an unreasonable choice as most multilingual computational models have been developed from an English perspective. However, it poses limitations on the interpretation of the linguistic results. For future work, we propose taking a more multilingual perspective. It should also be noted that our methods cannot capture phonetic or phonological changes such as vowel shift. We understand our proposed methodology as an addition to the inventory of linguistic analysis, not as a replacement.

We introduced a methodology to analyze the semantic structure of multilingual distributional representations. Our method is inspired by research in neuroscience on comparing computational representations to human brain data. We adapted the analysis to compare representations across language families. We show that our method can be used for phylogenetic reconstruction and that it captures subtle semantic differences of words between language families. In addition, we proposed a new measure for identifying phenomena of semantic drift. Our qualitative examples indicate that this measure can generate new hypotheses for comparative linguistics.

The computational models for sentences are available for a huge range of languages. In this article, we restricted the languages to those used in previous work for a reasonable comparison. We now plan to corroborate our findings on the whole spectrum and to further extend the word-based analyses of semantic drift.

Bulgarian (bg), Catalan (ca), Croatian (hr), Czech (cs), Danish (da), Dutch (nl), English (en), Estonian (et), Finnish (fi), French (fr), German (de), Greek (el), Hebrew (he), Hungarian (hu), Indonesian (id), Italian (it), Macedonian (mk), Norwegian (no), Polish (pl), Portuguese (pt), Romanian (ro), Russian (ru), Slovakian (sk), Slovenian (sl), Spanish (es), Swedish (sv), Turkish (tr), Ukrainian (uk)

Bulgarian (bg), Czech (cs), Danish (da), Dutch (nl), English (en), French (fr), German (de), Italian (it), Latvian (lv), Lithuanian (lt), Polish (pl), Portuguese (pt), Romanian (ro), Slovakian (sk), Slovenian (sl), Spanish (es), Swedish (sv)

all, and, animal, ashes, at, back, bad, bark, because, belly, big, bird, bite, black, blood, blow, bone, breast, breathe, burn, child, cloud, cold, come, correct, count, cut, day, die, dig, dirty, dog, drink, dry, dull, dust, ear, earth, eat, egg, eye, fall, far, fat, father, fear, feather, few, fight, fingernail, fire, fish, five, float, flow, flower, fly, fog, foot, forest, four, freeze, fruit, full, give, good, grass, green, guts, hair, hand, he, head, hear, heart, heavy, here, hit, hold, horn, how, hunt, husband, I, ice, if, in, kill, knee, know, lake, laugh, leaf, left, leg, lie, live, liver, long, louse, man, many, meat, moon, mother, mountain, mouth, name, narrow, near, neck, new, night, nose, not, old, one, other, play, pull, push, rain, red, right, river, road, root, rope, rotten, round, rub, salt, sand, say, scratch, sea, see, seed, sew, sharp, short, sing, sit, skin, sky, sleep, small, smell, smoke, smooth, snake, snow, some, spit, split, squeeze, stab, stand, star, stick, stone, straight, suck, sun, swell, swim, tail, that, there, they, thick, thin, think, this, three, throw, tie, tongue, tooth, tree, turn, two, vomit, walk, warm, wash, water, we, wet, what, when, where, white, who, wide, wife, wind, wing, wipe, with, woman, worm, year, yellow, you

ability, accomplished, angry, apartment, applause, argument, argumentatively, art, attitude, bag, ball, bar, bear, beat, bed, beer, big, bird, blood, body, brain, broken, building, burn, business, camera, carefully, challenge, charity, charming, clothes, cockroach, code, collection, computer, construction, cook, counting, crazy, damage, dance, dangerous, deceive, dedication, deliberately, delivery, dessert, device, dig, dinner, disease, dissolve, disturb, do, doctor, dog, dressing, driver, economy, election, electron, elegance, emotion, emotionally, engine, event, experiment, extremely, feeling, fight, fish, flow, food, garbage, gold, great, gun, hair, help, hurting, ignorance, illness, impress, invention, investigation, invisible, job, jungle, kindness, king, lady, land, laugh, law, left, level, liar, light, magic, marriage, material, mathematical, mechanism, medication, money, mountain, movement, movie, music, nation, news, noise, obligation, pain, personality, philosophy, picture, pig, plan, plant, play, pleasure, poor, prison, professional, protection, quality, reaction, read, relationship, religious, residence, road, sad, science, seafood, sell, sew, sexy, shape, ship, show, sign, silly, sin, skin, smart, smiling, solution, soul, sound, spoke, star, student, stupid, successful, sugar, suspect, table, taste, team, texture, time, tool, toy, tree, trial, tried, typical, unaware, usable, useless, vacation, war, wash, weak, wear, weather, willingly, word

Concepts in the database have German identifiers:

Auge::N, Ohr::N, Nase::N, Mund::N, Zahn::N, Zunge::N, Lippe::N, Stirn::N, Haar::N, Bart::N, Hals::N, Kopf::N, Rücken::N, Bauch::N, Brust::N, Arm::N, Ellenbogen::N, Hand::N, Finger::N, Knie::N, Oberschenkel::N, Bein::N, KöPrper::N, Haut::N, Blut::N, Ader::N, Sehne::N, Herz::N, Hunger::N, Träne::N, Geschmack::N, Geruch::N, Schlaf::N, Traum::N, Sonne::N, Mond::N, Stern::N, Luft::N, Wind::N, Welle::N, Wasser::N, Stein::N, Boden::N, Erde::N, Staub::N, Rauch::N, Feuer::N, Licht::N, Schatten::N, Wetter::N, Wolke::N, Schnee::N, Eis::N, Frost::N, Kälte::N, Donner::N, Schaum::N, See::N, Wiese::N, Wald::N, Hägel::N, Berg::N, Gipfel::N, Hö::N, Quelle::N, Bach::N, Fluss::N, Ufer::N, Meer::N, Bucht::N, Insel::N, Blume::N, Gras::N, Wurzel::N, Baum::N, Stamm::N, Rinde::N, Ast::N, Zweig::N, Birke::N, Kiefer[Baum]::N, Tanne::N, Horn::N, Feder::N, Fell::N, Flügel::N, Kralle::N, Schwanz::N, Ei::N, Nest::N, Bau::N, Kuh::N, Bulle::N, Pferd::N, Schaf::N, Schwein::N, Elch::N, Fuchs::N, Hase::N, Maus::N, Wolf::N, Vogel::N, Schwarm::N, Huhn::N, Gans::N, Adler::N, Ente::N, Eule::N, Krä::N, Kuckuck::N, Fisch::N, Spinne::N, Ameise::N, Mücke::N, Fliege::N, Schmetterling::N, Beere::N, Apfel::N, Korn::N, Heu::N, Grube::N, Spur::N, Asche::N, Dreck::N, Gold::N, Silber::N, Glas::N, Lehm::N, Sand::N, Kind::N, Familie::N, Eltern::N, Mutter::N, Sohn::N, Bruder::N, Schwester::N, Onkel::N, Ehefrau::N, Freude::N, Wunsch::N, Gedanke::N, Verstand::N, Sinn::N, Grund::N, Wahrheit::N, Gespräch::N, Erzählung::N, Neuigkeit::N, Sprache::N, Stimme::N, Wort::N, Zeichen::N, Laut::N, Ton::N, Lied::N, Ruhe::N, Leute::N, Volk::N, Arbeit::N, Gast::N, Geschenk::N, Spiel::N, Freund::N, Angelegenheit::N, Anzahl::N, Art::N, Stäck::N, Teil::N, Hälfte::N, Kreis::N, Kreuz::N, Linie::N, Entfernung::N, Platz::N, Ort::N, Seite::N, Mitte::N, Gegenstand::N, Sache::N, Rand::N, Kante::N, Ecke::N, Spitze::N, Ende::N, Loch::N, Winkel::N, Muster::N, Länge::N, Gewicht::N, Reihe::N, Last::N, Norden::N, Süden::N, Osten::N, Holz::N, Brett::N, Stock::N, Stab::N, Rohr::N, Haus::N, Heim::N, Ofen::N, Fußboden::N, Stuhl::N, Wiege::N, Bett::N, Tür::N, Zaun::N, Dach::N, Besen::N, Haken::N, Griff::N, Bild::N, Figur::N, Puppe::N, Kessel::N, Essen::N, Brot::N, Butter::N, Öl::N, Salz::N, Suppe::N, Honig::N, Milch::N, Leder::N, Wolle::N, Stoff::N, Nadel::N, Faden::N, Knopf::N, Hemd::N, Kragen::N, Gürtel::N, Ring::N, Band::N, Spiegel::N, Kraft::N, Stärke::N, Krankheit::N, Wunde::N, Arznei::N, Brücke::N, Brunnen::N, Weide::N, Pfad::N, Weg::N, Straße::N, Dorf::N, Stadt::N, Brief::N, Buch::N, Leben::N, Tod::N, Grab::N, Kirche::N, Sünde::N, Gott::N, Chef::N, Arzt::N, Geld::N, Preis::N, Ware::N, Nutzen::N, Reichtum::N, Welt::N, König::N, Macht::N, Grenze::N, Krieg::N, Gewalt::N, Kampf::N, Bogen[Waffe]::N, Lüge::N, Schaden::N, Schuld::N, Alter::N, Schluss::N, Zeit::N, Tag::N, Morgen::N, Nacht::N, Woche::N, Monat::N, Jahr::N, Sommer::N, Herbst::N, Winter::N, Januar::N, Februar::N, März::N, April::N, Mai::N, Juni::N, Juli::N, August::N, Oktober::N, November::N, Dezember::N, Samstag::N, Sonntag::N, groß::A, klein::A, kurz::A, schmal::A, dicht::A, dick[Gegenstand]::A, dünn::A, fein::A, fest::A, glatt::A, hart::A, rund::A, schön::A, warm::A, kalt::A, kühl::A, nass::A, voll::A, geschlossen::A, roh::A, reif::A, süß::A, bitter::A, sauer::A, hell::A, dunkel::A, weiß::A, gelb::A, grün::A, grau::A, wertvoll::A, blind::A, taub::A, stark::A, schlank::A, faul::A, lustig::A, nackt::A, gut::A, richtig::A, alt::A, neu::A, alt[Lebewesen]::A, jung::A, arm::A, reich::A, bekannt::A, berühmt::A, fremd::A, linker::A, rechter::A, erster::A, dritter::A, letzter::A, zusammen::ADV, jetzt::ADV, dann::ADV, immer::ADV, hier::ADV, sehr::ADV, so::ADV, noch::ADV, schon::ADV, zwischen::PRP, dies::PRN, das::PRN, alles::PRN, ich::PRN, du::PRN, er::PRN, wir::PRN, sie::PRN, wer::FPRN, wo::FADV, wie::FADV, und::CNJ, oder::CNJ, fallen::V, wachsen::V, atmen::V, trinken::V, essen::V, sterben::V, springen::V, gehen::V, kommen::V, finden::V, vorbereiten::V, sehen::V, tun::V, können::V, stellen::V, machen::V, bauen::V, reparieren::V, tragen::V, teilen::V, erhalten::V, nehmen::V, wählen::V, bewahren::V, leiten::V, einladen::V, singen::V, tanzen::V, verteidigen::V, sammeln::V, hüten::V, sprechen::V, bitten::V, übersetzen::V, glauben::V, wissen::V, lesen::V, schreiben::V, besitzen::V, kaufen::V

The work presented here was funded by the Netherlands Organisation for Scientific Research (NWO), through a Gravitation Grant 024.001.006 to the Language in Interaction Consortium. We gratefully acknowledge Bas Cornelissen and Tom Lentz for valuable discussions of earlier versions of the article. We would like to thank the anonymous reviewers for their very constructive and helpful feedback and their attention to detail.