What Do Language Representations Really Represent?

Words can be represented with distributed word representations, currently often in the form of word embeddings. Similarly to how words can be embedded, so can languages, by associating each language with a real-valued vector known as a language representation, which can be used to measure similarities between languages. This type of representation can be obtained by, for example, training a multilingual model for some NLP task (Johnson et al. 2017; Malaviya, Neubig, and Littell 2017; Östling and Tiedemann 2017). The focus of this work is on the evaluation of similarities between such representations. This is an important area of work, as computational approaches to typology (Dunn et al. 2011; Cotterell and Eisner 2017; Bjerva and Augenstein 2018) have the potential to answer research questions on a much larger scale than traditional typological research (Haspelmath 2001). Furthermore, having knowledge about the relationships between languages can help in NLP applications (Ammar et al. 2016), and having incorrect interpretations can be detrimental to multilingual NLP efforts. For instance, if the similarities between languages in an embedded language space were to be found to encode geographical distances (Figure 1), any conclusions drawn from use of these representations would not likely be of much use for most NLP tasks. The importance of having deeper knowledge of what such representations encapsulate is further hinted at by both experiments with interpolation of language vectors (Östling and Tiedemann 2017), as well as multilingual translation models (Johnson et al. 2017).

Several previous authors have done preliminary investigations into the structure of language representations: Östling and Tiedemann (2017), Malaviya, Neubig, and Littell (2017), and Johnson et al. (2017) in the context of language modeling and machine translation, all of them using multilingual data. In this work we follow up on the findings of Rabinovich, Ordan, and Wintner (2017), who, by using language representations consisting of manually specified feature vectors, find that the structure of a language representation space is approximately preserved by translation. However, their analysis only stretches as far as finding a correlation between their language representations and genetic distance, even though the latter is correlated to several other factors. We apply a multilingual language model to this problem, and evaluate the learned representations against a set of three language properties: (i) genetic distance (families), (ii) a novel measure of syntactic similarity (structural), and (iii) distance of language communities (geographical). We investigate:

• RQ1. In what way do different language representations encode language similarities? In particular, is genetic similarity what is really captured?

• RQ2. What causal relations can we find between language representation similarities?

Our work is most closely related to Rabinovich, Ordan, and Wintner (2017), who investigate representation learning on monolingual English sentences, which are translations from various source languages to English from the Europarl corpus (Koehn 2005). They use a feature-engineering approach to predict source languages and learn an Indo-European family tree using their language representations, showing that there are significant traces of the source languages in translations. They use features based on sequences of part-of-speech (POS) tags, function words, and cohesive markers. Additionally, they posit that the similarities found between their representations encode the genetic relationships between languages. We show that this is not the strongest explanation of the similarities, as a novel syntactic measure offers far more explanatory value, which we further substantiate by investigating causal relationships between language representations and similarities (Pearl 2009). This is an important finding as it highlights the need for thoroughly substantiating linguistic claims made based on empirical findings. Further, understanding what similarities are encoded in language embeddings provides insights into how language embeddings could be used for downstream multilingual NLP tasks. If language representations are used for transfer learning to low-resource languages, having an incorrect view of the structure of the language representation space can be dangerous. For instance, the standard assumption of genetic similarity would imply that the representation of the Gagauz language (Turkic, spoken mainly in Moldova) should be interpolated from the genetically very close Turkish, but this would likely lead to poor performance in syntactic tasks because the two languages have diverged radically in syntax relatively recently.

Figure 2 illustrates the data and problem we consider in this paper. We are given a set of English gold-standard translations from the official languages of the European Union, based on speeches from the European Parliament.¹ We wish to learn language representations based on these data, and investigate the linguistic relationships that hold between the resulting representations (RQ1). It is important to abstract away from the surface forms of the translations as, for example, speakers from certain regions will tend to talk about the same issues, or places. We therefore introduce three levels of abstraction: (i) training on function words and POS; (ii) training on only POS tags (POS in Figure 2); (iii) training on sequences of dependency relation tags (DepRel in Figure 2), and constituent tags. This annotation is obtained using UDPipe (Straka, Hajic, and Straková 2016).

For each level of abstraction, we train a multilingual neural language model in order to obtain representations (vectors in ℝ^k) that we can analyze further (RQ1). Note that this model is multilingual in the sense that we model the source language of each input sequence, whereas the input sequences themselves are, for example, sequences of POS tags. Our model is a multilingual language model using a standard two-layer long short-term memory architecture. Multilinguality is approached similarly to Östling and Tiedemann (2017), who include a language representation at each time-step. That is to say, each input is represented both by a symbol representation, c, and a language representation, l ∈ L. Because the set of language representations L is updated during training, the resulting representations encode linguistic properties of the languages. Whereas Östling and Tiedemann (2017) model hundreds of languages, we model only English—however, we redefine L to be the set of source languages from which our translations originate.

We now consider the language representations obtained from training our neural language model on the input sequences with different representations of the text (characters, POS sequences, etc.). We cluster the language representations—vectors in ℝ^k—hierarchically² and compute similarities between our generated trees and the gold tree of Serva and Petroni (2008), using the distance metric from Rabinovich, Ordan, and Wintner (2017).³ Our generated trees yield comparable results to previous work (Table 1).

Our first experiments deal with training directly on the raw translated texts. This is likely to bias representations by speakers from different countries talking about specific issues or places (as in Figure 2), and gives the model comparatively little information to work with as there is no explicit syntactic information available. As a consequence of the lack of explicit syntactic information, it is unsurprising that the results (LM-Raw in Table 1) only marginally outperform the random baseline.

To abstract away from the content and negate the geographical effect we train a new model on only function words and POS. This performs almost on par with LM-Raw (LM-Func in Table 1), indicating that the level of abstraction reached is not sufficient to capture similarities between languages. We next investigate whether we can successfully abstract away from the content by removing function words, and only using POS tags (LM-POS in Table 1). Although Rabinovich, Ordan, and Wintner (2017) produce sensible trees by using trigrams of POS and function words, we do not obtain such trees in our most similar settings. One hypothesis for why this is the case is the differing architectures used—indicating that our neural architecture does not pick up on the trigram-level statistics present in their explicit feature representations.

To force the language model to predict as much syntactic information as possible, we train on bracketed phrase structure trees. Note that this is similar to the target side of Vinyals et al. (2015). All content words are replaced by POS tags, and function words are kept. This results in a vocabulary of 289 items (phrase and POS tags and function words). Syntactic information captures more relevant information for reconstructing trees than previous settings (LM-Phrase in Table 1), yielding trees of similar quality to previous work.

We also compare to the Universal Dependencies (UD) formalism, as we train the language model on tuples encoding the dependency relation and POS tag of a word, the head direction, and the head POS tag (LM-Deprel in Table 1). The LM-Phrase and LM-Deprel models yield the best results overall, due to their having access to higher levels of abstraction via syntax. The fact that sufficient cues for the source languages can be found here shows that source language affects the grammatical constructions used (cf. Gellestam 1986).

Our main contribution is to investigate whether genetic distance between languages is captured by language representations, or if other distance measures provide more explanation (RQ1). Having shown that our language representations can reproduce genetic trees on par with previous work, we now compare the language embeddings using three different types of language distance measures: genetic distance estimated by methods from historical linguistics, geographical distance of speaker communities, and a novel measure for the structural distances between languages.

We rely on the coordinates provided by Glottolog (Hammarström, Forkel, and Haspelmath 2017). These are by necessity approximate, because the geography of a language cannot accurately be reduced to a single point denoting the geographical center point of where its speakers live. Still, this provides a way of testing the influence of geographical factors such as language contact or political factors affecting the education system.

To summarize the structural properties of each language, we use counts of dependency links from the UD treebanks, version 2.1 (Nivre et al. 2017). Specifically, we represent each link by combining head and dependent POS, dependency type, and direction. This yields 8,607 combinations, so we represent each language by a 8,607-dimensional normalized vector, and compute the cosine distance between these language representations.

Figure 3 shows the result of clustering these vectors (Ward clustering, cosine distance). Although strongly correlated with genealogical distance, significant differences can be observed. Romanian, as a member of the Balkan sprachbund, is distinct from the other Romance languages. The North Germanic (Danish, Swedish) and West Germanic (Dutch, German) branches are separated through considerable structural differences, with English grouped with the North Germanic languages despite its West Germanic origin. The Baltic languages (Latvian, Lithuanian) are grouped with the nearby Finnic languages (Estonian, Finnish) rather than their distant Slavic relatives.

This idea has been explored previously by Chen and Gerdes (2017), who use a combination of relative frequency, length, and direction of deprels. We, by comparison, achieve an even richer representation by also taking head and dependent POS into account.

Although we are able to reconstruct phylogenetic language trees in a similar manner to previous work, we wish to investigate whether genetic relationships between languages really is what our language representations represent.

We generate distance matrices A_ρ, where each entry a_i,j represents the ρ-similarity between the i^th and j^th languages, using the three similarity measures outlined in §4. Then, the entries in A_gen contain pairwise genetic distances, computed by summing the weights of all edges on the shortest path between two leaves (languages). Similarly, the entries in A_geo contain the geographical distance between countries associated with the languages. Lastly, the entries in A_struct contain the cosine distance between the language representations, which are encoded in 8,607-dimensional normalized vectors.

Figure 4 shows the Spearman correlation coefficients between each pair of these matrices. The strongest correlations can be found between the language embeddings, showing that they have similar representations. The correlations between our three distance measures are also considerable (e.g., between geographical and structural distances). This is expected, as languages that are close to one another geographically tend to be similar due to language contact, and potentially shared origins (Velupillai 2012).

Most interestingly, the language embedding similarities correlate the most strongly with the structural similarities, rather than the genetic similarities, thus answering RQ1. Although previous work by Rabinovich, Ordan, and Wintner (2017) has shown that relatively faithful phylogenetic trees can be reconstructed, we have found an alternative interpretation to these results with much stronger similarities to structural similarities. This indicates that, as often is the case, although similarities between two factors can be found, this is not necessarily the factor with the highest explanatory value (Roberts and Winters 2013).

We further strengthen our analysis by investigating RQ2, looking at the relationships between our variables in a Causal Network (Pearl 2009). We use a variant of the Inductive Causation algorithm, namely, IC* (Verma and Pearl 1992). It takes a distribution as input, and outputs a partially directed graph that denotes the (potentially) causal relationships found between each node in the graph. Here, the nodes represent our similarity measures and language embedding distances. The edges in the resulting graph can denote genuine causation (unidirectional edges), potential causation (dashed unidirectional edges), spurious associations (bidirectional edges), and undetermined relationships (undirected edges) (Pearl 2009). Running the algorithm on our distribution based on all the distance measures and language embeddings from this work yields a graph with the following properties, as visualized in Figure 5.⁴

We observe two clusters, marking associations between distance measures and language representations. Interestingly, the only link found between the clusters is an association between the structural similarities and our raw model. This further strengthens our argument, as the fact that no link is found to the genetic similarities shows that our alternative explanation has higher explanatory value, and highlights the need for controlling for more than a single linguistic factor when seeking explanations for one’s results.

We train language representations on three levels of syntactic abstraction, and explore three different explanations to what language representations represent: genetic, geographical, and structural distances. On the one hand, we extend on previous work by showing that phylogenetic trees can be reconstructed using a variety of language representations (Rabinovich, Ordan, and Wintner 2017). On the other, contrary to a claim of Rabinovich, Ordan, and Wintner (2017), we show that structural similarities between languages are a better predictor of language representation similarities than genetic similarities. As interest in computational typology is increasing in the NLP community (Östling 2015; Bjerva and Augenstein 2018; Gerz et al. 2018; Ponti et al. 2018), we advocate for the necessity of explaining typological findings through comparison.