Unsupervised Word Translation with Adversarial Autoencoder

Learning crosslingual word embeddings has been shown to be an effective way to transfer knowledge from one language to another for many key linguistic tasks, including machine translation, named entity recognition, part-of-speech tagging, and parsing (Ruder, Vulic, and Sogaard 2017). Whereas earlier efforts used large parallel corpora to solve the associated word alignment problem (Luong, Pham, and Manning 2015), broader applicability demands methods to relax this requirement because acquiring a large corpus of parallel data is not feasible in most scenarios. Recent methods instead use embeddings learned from monolingual corpora, and then learn a linear mapping from one language to another with the underlying assumption that two embedding spaces exhibit similar geometric structures, also known as the isomorphic assumption. This allows the model to learn effective crosslingual representations without expensive supervision.

Given monolingual word embeddings of two languages, Mikolov, Le, and Sutskever (2013) show that a linear mapping can be learned from a seed dictionary of 5,000 word pairs by minimizing the sum of squared Euclidean distances between the mapped vectors and the target vectors. Subsequent studies (Xing et al. 2015; Artetxe, Labaka, and Agirre 2016, 2017; Smith et al. 2017) propose to improve the model by normalizing the embeddings, imposing an orthogonality constraint on the mapper, and modifying the objective function. These methods assume some supervision in the form of a seed dictionary, although recently fully unsupervised methods have shown competitive results. Zhang et al. (2017a, 2017b) first reported encouraging results for unsupervised models with adversarial training. Conneau et al. (2018) improved this approach with post-mapping refinements, showing impressive results for many language pairs. Their learned mapping was then successfully used to train a fully unsupervised neural machine translation system (Lample et al. 2018a, 2018b).

Although successful, adversarial training has been criticized for not being stable and failing to converge, inspiring researchers to propose non-adversarial methods more recently (Xu et al. 2018; Hoshen and Wolf 2018; Alvarez-Melis and Jaakkola 2018; Artetxe, Labaka, and Agirre 2018b). In particular, Artetxe, Labaka, and Agirre (2018b) show that the adversarial methods of Conneau et al. (2018) and Zhang et al. (2017a, 2017b) fail for many difficult language pairs.

In this article, we revisit adversarial training and propose a number of key improvements that yield more robust training and improved mappings. Our main idea is to learn the crosslingual mapping in a projected latent space (code space) and add more constraints to guide the unsupervised mapping in this space. We accomplish this by proposing a novel adversarial autoencoder framework (Makhzani et al. 2015), where adversarial mapping is done at the latent code space as opposed to the original embedding space. This gives the model the flexibility to automatically induce the required geometric structures in its code space that could potentially yield better mappings. Figure 1 shows a conceptual demonstration of our main idea.

Søgaard, Ruder, and Vulić (2018) recently found that the isomorphic assumption made by most existing methods does not hold in general even for two closely related languages like English and German. In their words, “approaches based on this assumption have important limitations” (page 778). By performing nonlinear transformations of the original embeddings into their respective code spaces in the autoencoders and then mapping the latent codes of two languages through adversarial training, our approach therefore departs from the isomorphic assumption.

In our adversarial training, not only the mapper but also the target encoder is trained to fool the language discriminator. This forces the discriminator to improve its discrimination skills, which in turn pushes the mapper to generate indistinguishable translation. To guide the mapping, we include two additional constraints. Our first constraint enforces cycle consistency so that code vectors after being translated from one language to another, and then translated back to their source space, remain close to the original vectors. The second constraint ensures reconstruction of the original input word embeddings from the back-translated codes. This grounding step forces the model to retain word semantics during the mapping process and yields more stable training.

The initial bilingual dictionary induced by adversarial training (or any other unsupervised method) is generally of lower quality than what could be achieved by a supervised method. Conneau et al. (2018) and Artetxe, Labaka, and Agirre (2018b) propose fine-tuning methods to refine the initial mappings. In particular, Conneau et al. (2018) refine the initial mapping by iteratively solving the Procrustes problem and applying a dictionary induction step. Artetxe, Labaka, and Agirre (2018b) propose a multistep dictionary induction framework. Our work incorporates two types of refinement procedures, namely, refinement with Procrustes solution and refinement with symmetric re-weighting, a step proposed by Artetxe, Labaka, and Agirre (2018b). We perform refinement with the Procrustes solution in the code space, while refinement with symmetric re-weighting is done with the original word embeddings. This way our overall framework combines the two refinement procedures to get the best of both.

In order to demonstrate the effectiveness and robustness of our approach, we conduct a series of experiments with eight different language pairs (in both directions) comprising high- and low-resource languages from two different data sets. We also perform extensive ablation studies to understand the contribution of different components of our adversarial autoencoder model and different refinement procedures. Our main findings are the following.

• (i) 

  Our adversarial method is more robust and yields significant gains over the adversarial method of Conneau et al. (2018) for all translation tasks in all evaluation measures.

• (ii) 

  Our method with adversarial autoencoder exhibits better performance than other existing supervised and unsupervised methods in most of the translation tasks.

• (iii) 

  The ablation study of our adversarial autoencoder model reveals that cycle consistency contributes the most, while adversarial training of the target encoder and post-cycle reconstruction also have significant effects.

• (iv) 

  The in-depth analysis of the refinement procedures shows that symmetric re-weighting is a powerful method and complements the Procrustes solution based refinement method.

We have released our source code at https://ntunlpsg.github.io/project/unsup-word-translation/.

In the rest of the article, we first review the related supervised and unsupervised (both adversarial and non-adversarial) word translation models in Section 2, then present our proposed unsupervised approach with adversarial autoencoder and refinement procedures in Section 3. In Section 4, we present the experimental settings—the data sets, and the supervised and the unsupervised baselines that we compare with. We present our results with in-depth analysis in Section 5. Finally, we summarize our contributions with future directions in Section 6.

In recent years, a number of methods have been proposed to learn a bilingual dictionary from monolingual word embeddings.¹ Many of these methods use an initial seed dictionary. However, more recently, researchers have attempted to eliminate the seed dictionary totally and learn the mapping in a purely unsupervised way. In this section, we give an overview of existing supervised and unsupervised word translation methods. We also discuss the hubness problem that often occurs in these methods and approaches to alleviate the effect of this problem.

Instead of learning a mapping from the source to the target embedding space, Faruqui and Dyer (2014) use a technique based on Canonical Correlation Analysis to project both source and target embeddings to a common low-dimensional space, where the correlation of the word pairs in the seed dictionary is maximized.

In their follow-up work, Artetxe, Labaka, and Agirre (2017) obtain competitive results using a seed dictionary of only 25 word pairs. They propose a self-learning framework that performs two steps iteratively until convergence. In the first step, they use the dictionary (starting with the seed) to learn a linear mapping, which is then used in the second step to induce a new dictionary.

In their more recent work, Artetxe, Labaka, and Agirre (2018a) propose a multistep framework that generalizes previous studies. Their framework consists of several steps: whitening, orthogonal mapping, re-weighting, de-whitening, and dimensionality reduction. They show that existing methods can be explained in terms of these steps. For example, regression methods such as the method of Mikolov, Le, and Sutskever (2013) correspond to the case where whitening is applied to both the source and target language embeddings, re-weighting is applied only to source language embeddings, and de-whitening is applied to both language embeddings. The canonical method of Faruqui and Dyer (2014) corresponds to the case where whitening is applied to both the source and target language embeddings, and dimensionality reduction is applied to both, but re-weighting and de-whitening are not performed. Similarly, orthogonal methods such as the methods of Artetxe, Labaka, and Agirre (2016) and Smith et al. (2017) correspond to the case where only orthogonal mapping is applied.

As mentioned, a more recent line of research attempts to eliminate the seed dictionary and learn the bilingual mapping in a completely unsupervised way. Initial approaches used adversarial methods, but some non-adversarial methods have also been proposed more recently.

Barone (2016) is the first to propose a model for solving bilingual word translation in an unsupervised way. He initially used an adversarial network similar to Conneau et al. (2018), and found that the mapper (which is also the encoder) translates everything to a single embedding, commonly known as the mode collapse issue (Goodfellow 2017). To preserve diversity in mapping, he then used a decoder to reconstruct the source embedding from the mapped embedding, extending the framework to an adversarial autoencoder. His analysis (qualitative) shows promising but not competitive with methods that use bilingual seeds. He suspected issues with adversarial training and with the underlying isomorphic assumption.

In our work, we successfully address these issues with an improved framework that also relaxes the isomorphic assumption. Our framework comprises two separate autoencoders, one for each language, which allows us to put more constraints to guide the mapping through adversarial training. We also distinguish the role of an encoder from the role of a mapper. The encoder projects embeddings to latent code vectors, which are then translated by the mapper.

Zhang et al. (2017a) first show encouraging results on unsupervised word translation with adversarial methods. They propose the following three unsupervised models.

• (i) 

  Unidirectional transformation model: This model is similar in spirit to the model of Conneau et al. (2018). The generator tries to transform the source embeddings such that they are indistinguishable from the target embeddings, whereas the discriminator tries to distinguish the real target embeddings from the ones that are generated by the generator.

• (ii) 

  Bidirectional transformation model: There are two generators in this model that transform embeddings from one language space to another language space. Two separate discriminators for each language are used to distinguish the real embeddings from the transformed ones.

• (iii) 

  Adversarial autoencoder model: In this model, the generator is responsible for transforming source embeddings to target space not only to make them indistinguishable by the discriminator but also for back-translating them to the source space. For this reason, they introduce reconstruction loss. Although they call it an adversarial autoencoder model, their approach is similar to the cycle GAN (Zhu et al. 2017).

Conneau et al. (2018) is the first to show impressive results for unsupervised word translation by pairing adversarial training with effective refinement methods. Given two monolingual word embeddings, their adversarial training plays a two-player game, where a linear mapper (generator) plays against a discriminator (Goodfellow et al. 2014). The discriminator is trained to distinguish between the original target embedding and the mapped source embedding in the target space. On the other hand, the mapper is jointly trained to fool the discriminator. They also impose the orthogonality constraint on the mapper to preserve the monolingual quality of the embeddings and to make the training more stable. After adversarial training, they extract a synthetic dictionary from the resulting shared embedding space. To ensure a high-quality synthetic dictionary, they consider the most frequent words and retain only mutual nearest neighbors in the dictionary induction process. For fine-tuning the linear mapper, they use the Procrustes solution in Equation (3), which is the closed form solution of Equation (1) under the orthogonality constraint. Similar to Artetxe, Labaka, and Agirre (2017), they do the fine-tuning in an iterative way. Because their method is purely unsupervised, instead of using any bilingual dictionary, they introduce an unsupervised selection metric for selecting the best model. This metric is highly correlated with the mapping quality and quantifies the closeness of the source and target embedding spaces. They use it as a stopping criterion as well as to select the best hyperparameters of the model.

Instead of using an adversarial loss, Xu et al. (2018) use Sinkhorn distance, another distributional similarity measure. They optimize the linear mapping in both directions (source to target and vice versa) for each language pair in the way that the source embeddings mapped to the target space match the distribution of the target embeddings. Moreover, when the mapped source embeddings from the target space are back-translated to the source space, they are maximally close to the original source embeddings, also known as cycle consistency (Zhu et al. 2017). To avoid the problem of getting stuck in a poor local minima, their model requires a good initial setting of the parameters. To ensure this, in the first phase of the training, they optimize the Wasserstein distance instead of Sinkhorn distance.

Similar to Zhang et al. (2017a) and Xu et al. (2018), we also incorporate cycle consistency along with the adversarial loss to train our adversarial autoencoder. However, whereas all the existing methods learn the mapping in the original embedding space, our approach learns it in the latent code space, considering both the mapper and the target encoder as adversarial with the discriminator. In addition, we use a post-cycle reconstruction to guide the mapping.

Despite being successful, adversarial models have been criticized for instability and failing to converge, prompting researchers to investigate non-adversarial methods for unsupervised mapping of embeddings, as we summarize them here.

Artetxe, Labaka, and Agirre (2018b) learn an initial dictionary by exploiting the structural similarity of the embeddings and use a robust self-learning algorithm to improve it iteratively. Their proposed method consists of the following four sequential steps.

• (i) 

  Preprocessing: They length-normalize the embeddings, mean-center along each dimension, and then length-normalize them again to ensure the unit length of the embeddings.

• (ii) 

  Fully unsupervised initialization: To create an initial dictionary, they follow the observation that in the similarity matrix of all words, each word has a different distribution of similarity values and equivalent words in two different languages have a similar “similarity” distribution. So under the strict isometric assumption, sorted similarity vectors of two equivalent words from two different languages will be the same. Based on this assumption, they induce an initial dictionary. In practice, the isometric assumption does not hold (Søgaard, Ruder, and Vulić 2018), thus resulting in a noisy dictionary.

• (iii) 

  Robust self-learning: This step iteratively improves the initial solution. Their goal is to learn two linear mappers to map the source and the target embeddings to a shared embedding space. Xie et al. (2018) show that under orthogonality constraint, mapping from the source to the target space is equivalent to mapping both source and target to a shared embedding space. In each step, their model computes the optimal dictionary over the similarity matrix after computing the orthogonal mapping. They propose the following key improvements in the dictionary induction step to make the self-learning more robust.

  • • 

    Stochastic dictionary induction: During dictionary induction, they randomly keep some elements in the similarity matrix with some probability, which enables the induced dictionary to vary from the current step to the next. This works as a drop-out method that should help the model to escape poor local optima. Although, in practice, they found that it does not make any difference for most of the language pairs.

  • • 

    Frequency based vocabulary cutoff: To keep the matrix size reasonable, they propose to consider only the most frequent words in the dictionary induction process.

  • • 

    CSLS retrieval: To mitigate the hubness problem (see Section 2.3), they use Cross-domain Similarity Local Scaling (CSLS) (Conneau et al. 2018) for finding the nearest neighbors.

  • • 

    Bidirectional dictionary induction: They propose to induce dictionaries in both directions (source to target and vice versa) and take their concatenation.

• (iv) 

  Final refinement through symmetric re-weighting: To improve the mapping further, they use a slightly modified version of their earlier multistep framework proposed in Artetxe, Labaka, and Agirre (2018a). In Section 3.2.2, we discuss this step in detail.

In our work, we use a refinement method that combines Procrustes solution and symmetric re-weighting. In contrast to existing methods, our Procrustes solution–based refinement works in the code space, while the symmetric re-weighting method works in the original embedding space. Our method combines the best of both refinement techniques in a way that is mutually advantageous.

Hoshen and Wolf (2018) observe that two sufficiently similar distributions can be aligned correctly with iterative matching methods. In their proposed method, they first align the second moment of the word distributions of two languages, and later refine the alignment iteratively. For aligning the second moment, they project the word vectors to the top P principal components using Principal Component Analysis assuming that some principal axes of variation are similar in many language pairs. Because the word distributions and components of variation are different in languages, projecting to the principal component does not align the languages in general. For this reason, they use a modified version of the Iterative Closest Point (ICP) method, which is popularly used in computer vision for 3D point cloud alignment. They call the method Mini-Batch Cycle ICP (MBC-ICP). This method learns the transformation from source to target space and vice versa. They use a cycle constraint to ensure that a word is transformed from one space to another and translated back to the original space without change. In the final step, they use fine-tuning similar to Conneau et al. (2018) by running the Procrustes solution iteratively.

Alvarez-Melis and Jaakkola (2018) cast the unsupervised embedding mapping problem as an optimal transport problem, and exploit the Gromov-Wasserstein distance, which measures how similarities between pairs of words relate across languages.

One problem that we have overlooked so far is how to find the nearest neighbor of a source word in the target space. Mikolov, Le, and Sutskever (2013) take the closest target embedding of the mapped source embedding in the target language space using cosine similarity as the similarity measure. Dinu, Lazaridou, and Baroni (2015) show that in high dimensional spaces this nearest neighbor finding approach leads to a detrimental phenomenon known as the hubness problem. Due to this problem, a few nodes (word embeddings) become hubs, whereas some others become anti-hubs. Hubs are the nodes that are nearest neighbors of many other nodes with high probability. On the other hand, anti-hubs are not nearest neighbors to any node.

To alleviate the hubness problem, Dinu, Lazaridou, and Baroni (2015) propose a method called the globally corrected neighbor retrieval method, where instead of returning the nearest neighbor of a (mapped) source embedding, it returns the target embedding for which the source embedding is the nearest neighbor, that is, it reverses the direction of the query. They solve ties by taking the candidate with the highest cosine similarity with the source embedding. Artetxe, Labaka, and Agirre (2016) termed this approach as inverted nearest neighbor retrieval.

Smith et al. (2017) combat the hubness problem by introducing inverted softmax method, which is built on the work of Dinu, Lazaridou, and Baroni (2015), and also works by reversing the direction of the query. To find the nearest neighbor, they use the softmax function instead of cosine in the similarity computations.

Among the existing solutions to penalize the similarity scores of hubs, CSLS generally performs better, and has become the standard measure of similarity search. In our approach, we also use CSLS for finding crosslingual nearest neighbors.

Let 𝒳 = {x₁, …, x_n} and 𝒴 = {y₁, …, y_m} be two sets consisting of n and m word embeddings of d-dimensions for a source and a target language, respectively. We assume that 𝒳 and 𝒴 are trained independently from monolingual corpora. Our aim is to learn a mapping f(x) in an unsupervised way (i.e., no bilingual dictionary given) such that for every x_i, f(x) corresponds to its translation in 𝒴. Figure 2 shows our overall approach, which has the same sequence of steps as Conneau et al. (2018). More specifically, the steps are:

• (i) 

  Induction of seed dictionary through adversarial training.

• (ii) 

  Iterative refinement of the initial mapping.

• (iii) 

  Application of CSLS for nearest neighbor search.

We propose a novel adversarial autoencoder to learn the initial mapping for inducing a seed dictionary (Section 3.1), and we incorporate existing refinement methods for steps (ii) and (iii) (Section 3.2). Without loss of generality, we use X ∈ ℝ^n×d and Y ∈ ℝ^m×d to denote the matrices containing the word embeddings of the source and target, respectively. Table 1 summarizes all the notations used throughout the article.

Let q(z_x|x)and q(z_y|y) be the encoding distributions of the two autoencoders. We use adversarial training to find a mapping between q(z_x|x) and q(z_y|y). This is in contrast with most existing methods (e.g., Conneau et al. 2018; Artetxe, Labaka, & Agirre 2017) that directly map the distribution of the source word embeddings p(x) to the distribution of the target p(y). As Søgaard, Ruder, and Vulić (2018) pointed out, the isomorphism does not hold in general between the word embedding spaces of two languages. Mapping the latent codes of two languages gives our model more flexibility to induce the required semantic structures in its code space that could potentially yield more accurate mappings.

As shown in Figure 3, we include two linear mappingsG: 𝒵_x → 𝒵_y and F: 𝒵_y → 𝒵_x to project the code vectors (samples from q(.|.)) from one language to the other. In addition, we have two language discriminators, L_𝒳 and L_𝒴. The discriminators are trained to discriminate between the mapped codes and the encoded codes, and the mappers and respective target encoders are jointly trained to fool their respective discriminator. For example, in the case of mapping Z to 𝒵_y by G, the target encoder is E_𝒴. On the other hand, when F maps Z_y to 𝒵_x, E_𝒳 is the target encoder. This results in a three-player game, where the discriminator tries to identify the origin of a code, and the mapper and the respective target encoder act together to prevent the discriminator from succeeding by making the mapped vector and the encoded vector as similar as possible. In the following, we present the components of our framework.

Our discriminators have the same architecture as Conneau et al. (2018). Specifically, they are feed-forward networks with two hidden layers of size 2,048 and Leaky-ReLU activations. We apply dropout with a rate of 0.1 on the input to the discriminators. Instead of using 1 and 0, we also apply a smoothing coefficient (s = 0.2) in the discriminator loss.

Note that in our framework we consider both the mapper and the target encoder as generators. This is in contrast to existing adversarial methods, which do not use any autoencoder on the target side. The mapper and the target encoder team up to fool the discriminator. This forces the discriminator to improve its skill and vice versa for the generators, forcing them to produce indistinguishable codes through better mapping.

Adversarial training (introduced above) maps a “bag” of source embeddings to a “bag” of target embeddings, and, in theory, the mapper can match the target language distribution (Goodfellow 2017). However, mapping at the bag level is often insufficient to learn the individual word level mappings. In fact, there are an infinite number of possible mappings that can match the same target distribution. Thus to learn better mappings, we need to enforce more constraints to our objective.

Apart from improved mapping, both cycle consistency and reconstruction lead to more stable training in our experiments. Specifically, they help our training to converge and get around the mode collapse issue (Goodfellow 2017). Because the model now has to translate the mapped code back to the source code and reconstruct the original word embedding, the generators cannot get away by mapping all source codes to a single target code.

The encoders (E_𝒳 and E_𝒴) and the mappers (G and F) obtained from our adversarial training give a good initial bilingual dictionary. However, as shown later in our experiments (Section 5), the results with these initial mappings are inferior to the supervised methods. One reason is that with adversarial training, we do not consider the embeddings and the covariance of the features (columns of X, Y, or Z) globally. In the refinement step, our goal is to enrich the initial mapping by considering the global properties of the embedding spaces. Previous studies (Conneau et al. 2018; Artetxe, Labaka, and Agirre 2018b) have shown that refinement procedures after the initial mappings boost the results and make them better or on par with the supervised approach. In our work, we experiment with two types of refinement (or fine-tuning) procedures: (i) refinement with the Procrustes solution, and (ii) refinement with symmetric re-weighting.

Similar to Conneau et al. (2018), we first induce a seed dictionary using the learned encoders and mappers from our adversarial training. In order to find the nearest target word (y) of a source word (x) in the translated code space (𝒵_y), we use the CSLS measure in Equation (4). As described in Section 2.3, CSLS works better than simple cosine similarity in mitigating the hubness problem. It penalizes the nodes that are close to many other nodes in the translated space. To construct the seed dictionary, we compute CSLS for K₁ most frequent source (resp., target) words, and select the translation pairs that are mutual nearest neighbors, that is, we select x-y as a translation pair if and only if y is the nearest neighbor of x in 𝒵_y and x is the nearest neighbor of y in 𝒵_x.

We perform this fine-tuning process iteratively: Induce a new dictionary using CSLS on the newly learned mapping, then use the dictionary in Procrustes solution to improve the mapping parameters (see Algorithm 1). For convergence, we use the following criterion: if the difference between the average similarity scores of two successive iteration steps is less than a certain threshold (we use 10⁻⁶), then stop the refinement process. Note that unlike Conneau et al. (2018), who use original word embeddings; our refinement with Procrustes solution uses the latent codes for both the Procrustes solution and the dictionary induction.

In a different line of research, Artetxe, Labaka, and Agirre (2018a, 2018b) perform the refinement and dictionary induction steps by mapping both the source and target embeddings into a common space through orthogonal transformation. In addition to the Procrustes solution, they also use symmetric re-weighting to transform the original source and target embeddings to a common space. Because we apply Procrustes solution in the code space (𝒵_𝒳 and 𝒵_𝒴), to get the best of both representation types, in our approach we apply symmetric re-weighting to the original embeddings (𝒳 and 𝒴). This refinement works in three steps: (a) embedding whitening, (b) orthogonal mapping, and (c) embedding de-whitening.

With the transformed de-whitened embeddings, we can now measure the similarity between any x ∈ X_w and y ∈ Y_w, and thereby induce a synthetic dictionary by finding the nearest neighbor of x in 𝒴_w using CSLS Equation (4). During dictionary induction, we only consider the K₂ most frequent words from the source and the target languages and retain only the mutual nearest neighbors. This process is then iterated to refine the initial mappings and the induced dictionary. Algorithm 2 shows the whole process in pseudocode.

In our approach, we combine the two refinement methods to get the best of both—Procrustes solution on the code space and symmetric re-weighting on the original embedding space. We sequentially apply the two refinement procedures (Algorithms 1 and 2) by iteratively performing the same two steps for each method: Induce a synthetic dictionary and refine the mappers.

We first induce a seed dictionary from adversarial training by considering the K most frequent words. To optimize the G and F mappings, we apply the Procrustes solution using the induced dictionary. We then use CSLS to find a new synthetic dictionary, which in turn is used to refine the mappings. We continue this process on the code space until convergence.

After the convergence of the refinement procedure, we take the updated mappings. Using the learned encoders from adversarial training and the newly updated mappings, we induce a new seed dictionary to apply the symmetric re-weighting procedure. We follow the three steps in Section 3.2.2 to transform the original embeddings into a common space. From these transformed source and target embeddings, we induce a new dictionary using CSLS. We apply this refinement procedure iteratively on the original embedding space until convergence.

We present the training procedure of our model and the overall word translation process in Algorithm 3. We first pre-train the autoencoders separately on monolingual embeddings (Step 1). This pre-training is required to induce word semantics (and relations) in the latent code space.

We start adversarial training (Step 2) by updating the discriminators for n_critics5 times, each time with a random batch. Then we update the generators (the mapper and target encoder) on the adversarial loss. The mappers then go through two more updates, one for cycle consistency and another for post-cycle reconstruction. The autoencoders (encoder-decoder) in this stage get updated only on the post-cycle reconstruction loss. We also apply the orthogonalization update to the mappers following Conneau et al. (2018) with β = 0.01.

Our training setting is similar to Conneau et al. (2018), and we apply the similar pre- and post-processing steps. We use stochastic gradient descent with a batch size of 32, a learning rate of 0.1, and a decay of 0.95.

For selecting the best model, we use the unsupervised validation criterion proposed by Conneau et al. (2018), which correlates highly with the mapping quality. In this criterion, 10,000 most frequent source words along with their most probable translations in the target space are considered. We use CSLS to find the most probable translation of a source word. The average cosine similarity between these pseudo translations is considered as the validation metric for model selection. For refinement, we follow the methods described in Section 3.2.

Following the tradition, we evaluate our approach on word translation (a.k.a.bilingual lexicon induction) task, which measures the accuracy of the predicted dictionary to a gold standard dictionary. In the following, we describe the data sets and the baselines used in our experiments.

To demonstrate the effectiveness of our method, we evaluate our models on two different data sets. The first one is from Conneau et al. (2018), which consists of FastText monolingual embeddings of (d =) 300 dimensions (Bojanowski et al. 2017) trained on Wikipedia monolingual corpus and gold dictionaries for 110 language pairs.² To show the generality of different methods, we consider seven (7) different language pairs with fourteen (7 × 2 = 14) different translation tasks from high- and low-resource languages from different language families. In particular, we evaluate on English (En) from/to Spanish (Es), German (De), Italian (It), Finnish (Fi), Arabic (Ar), Malay (Ms), and Hebrew (He). Malay and Hebrew are generally considered as low-resource languages. We will refer to this data set as the Conneau data set.

We also evaluate on the more challenging data set of Dinu, Lazaridou, and Baroni (2015) and its subsequent extension by Artetxe, Labaka, and Agirre (2018a).³ This data set contains monolingual embeddings for English, Spanish, German, Italian, and Finnish. According to Artetxe, Labaka, and Agirre (2018b), existing adversarial methods often fail to produce meaningful results on this data set. English, Italian, and German embeddings were trained on WacKy crawling corpora using CBOW (Mikolov et al. 2013), while Spanish and Finnish embeddings were trained on WMT News Crawl and Common Crawl respectively. The CBOW vectors are also of 300 dimensions. We refer to this data set as the Dinu-Artetxe data set.

We compare our method with the unsupervised models of (i) Conneau et al. (2018), (ii) Artetxe, Labaka, and Agirre (2018b), (iii) Alvarez-Melis and Jaakkola (2018), (iv) Xu et al. (2018), and (v) Hoshen and Wolf (2018). To evaluate how our unsupervised method compares with methods that rely on a bilingual seed dictionary, we follow Conneau et al. (2018), and compute a supervised baseline that uses the Procrustes solution directly on the seed dictionary (5,000 pairs) to learn the mapping function, and then uses CSLS to do the nearest neighbor search. We refer to this baseline as Procrustes-CSLS.

We also compare with the supervised approaches of Artetxe, Labaka, and Agirre (2017, 2018a), which to our knowledge are the state-of-the-art supervised systems. For some of the baselines, results are reported from their papers, while for the rest we report results by running the publicly available codes on our machine.

For training our model on all language pairs, the weight for cycle consistency (λ₁) in Equation (16) was always set to 5, and the weight for post-cycle reconstruction (λ₂) was set to 1.⁴ The hidden layer dimensions of our encoder and decoder are set to 400 for both the first and second layer. We found the dimension of the code vectors to be crucial (especially for Arabic and low-resource languages), which we set through hyperparameter search. During the dictionary induction process of both refinement procedures, we consider 30,000 most frequent words (value of K₁ and K₂) from the source and target languages.

We present our results on high- and low-resource languages from Conneau and Dinu-Artetxe data sets in Tables 2–4. In each case, we present results for three different refinement procedures based on the same seed dictionary induced by our adversarial autoencoder: (i) refinement of Conneau et al. (2018) referred to as Conneau refinement, (ii) refinement of Artetxe, Labaka, and Agirre (2018b) referred to as Artetxe refinement, and (iii) our proposed refinement method that combines Procrustes solution and symmetric re-weighting. Through experiments and analysis, our goal is to assess the following.

• (i) 

  Does the unsupervised mapping method based on our proposed adversarial autoencoder model improve over the best existing adversarial method of Conneau et al. (2018) in terms of mapping accuracy and convergence (Section 5.1)?

• (ii) 

  How does our unsupervised mapping method compare with other unsupervised and supervised approaches (Section 5.2)?

• (iii) 

  Which components of our adversarial autoencoder model attribute to improvements (Section 5.3)?

• (iv) 

  What is the impact of different refinement methods in our approach (Section 5.4)?

Because our approach follows the same steps as Conneau et al. (2018), we first compare our proposed model with their model on their data set. Table 2 presents the results for Es, De, It, and Fi, and Table 3 presents the results for Ar, Ms, and He. In the tables, we present the numbers that they reported in their paper (Conneau et al. (2018) (paper)) as well as the results that we get by running their code on our machine (Conneau et al. (2018) (code)). For a fair comparison with respect to the quality of the learned mappings (or induced seed dictionary), in this subsection we only consider the results of our approach that use the refinement procedure of Conneau et al. (2018).

In Table 2, we see that our Adversarial autoencoder + Conneau refinement outperforms Conneau et al. (2018) in all the eight translation tasks. For Spanish, German, and Italian, the gains are in the range of 0.3% to 1.3%. The improvement is much higher (about 11%) for English to Finnish. Also notice that for Finnish to English, our method gives 64.0% word translation accuracy (P@1), whereas the method of Conneau et al. (2018) fails to converge for this task.

Our method is also superior to theirs for the Arabic and low-resource language pairs (Ms and He) in Table 3. Here our method gives consistent gains ranging from 2.3% to 4.5%. Note specifically that Malay (Ms) is a low-resource language, and FastText contains word vectors for only 155K Malay words. We found their model to be very fragile for En from/to Ms and does not converge at all for Ms → En. We ran their code 10 times for Ms → En but failed every time. Compared with that, our method is more robust and converged most of the time we ran.

If we compare our Adversarial autoencoder + Conneau refinement with Conneau et al. (2018) on the Dinu-Artetxe data set in Table 4, we see here also our method performs better than their method in all the eight translation tasks. In this data set, our method shows more robustness compared to their method. For example, their method had difficulties in converging for En from/to Es, De, and Fi translation tasks. For example, their model converges only 2 times out of 10 attempts for En → Es, while for Es → En, En ↔ De (both directions) and En ↔ Fi (both directions), it did not converge a single time in 10 attempts. Compared with that, our method was more robust and converged most of the time. In Section 5.3, we compare our model with Conneau et al. (2018) more rigorously by evaluating them with and without fine-tuning and measuring their performance on P@1, P@5, and P@10.

These comparisons with Conneau et al. (2018) using the same refinement method demonstrate that our approach of performing the crosslingual mapping in the code space is more effective and can learn more robust mapping functions. Figures 4a and 4b, respectively, show the t-SNE plots for English to Finnish translation on the Conneau data set for the model of Conneau et al. (2018) and our model. Notice that the English words and their Finnish translations are better mapped in our code space compared with the mappings in the original embedding space by Conneau et al. (2018).

In this section, we compare our model with other state-of-the-art methods that do not follow the same procedure as us and Conneau et al. (2018). For example, Artetxe, Labaka, and Agirre (2018b) do the initial mapping in the similarity space, then they apply a different refinement procedure on that.

Let us first consider the results on the Conneau data set in Table 2. Our method performs better than other methods in all eight translation tasks on this data set. Among the other unsupervised baselines, Artetxe, Labaka, and Agirre (2018b) exhibit better results than others. On the Dinu-Artetxe data set in Table 4, our model achieves better performance than most of the other methods. Only for En from/to Fi does the supervised model of Artetxe, Labaka, and Agirre (2018a) perform better than our method. As we mentioned earlier, this data set is more challenging, where other unsupervised methods except Artetxe, Labaka, and Agirre (2018b) fail to converge in most of the word translation tasks.

For Arabic and low-resource languages in Table 3, our model exhibits better performance than others in three out of six translation tasks. Only the supervised model of Artetxe, Labaka, and Agirre (2018a) performs better than our method in the rest of the three translation tasks. Here our model gives consistent gains compared with other unsupervised models. For En ↔ Ms, we see the similar phenomenon that other unsupervised methods except Artetxe, Labaka, and Agirre (2018b) fail to converge.

We notice that the unsupervised method of Artetxe, Labaka, and Agirre (2018b) gives better results than other baselines. To understand whether the improvements of their method are due to a better initial mapping or better post-processing, we conducted two additional experiments. In our first experiment, we use their method to induce the initial seed dictionary and then apply iterative Procrustes solution for refinement. Table 5 shows the results. Surprisingly, on both data sets their initial mappings fail to produce any reasonable results. So we suspect that the main gain in Artetxe, Labaka, and Agirre (2018b) comes from their fine-tuning method, which they call robust self learning. In the second experiment, we use the initial dictionary induced by our adversarial training and then apply their refinement procedure. Here, for most of the translation tasks, we achieve better results; see the model Adversarial autoencoder + Artetxe refinement in Table 2–4. This shows that the initial dictionary generated by our model is better than their model.

We further analyze our adversarial autoencoder model by dissecting it and measuring the contribution of each novel component that is proposed in this work. We achieve this by incrementally removing a new component from our model and evaluating it on different translation tasks. In order to better understand the contribution of each component, we evaluate each model by measuring its P@1, P@5, and P@10 with fine-tuning and without fine-tuning. For fine-tuning, we use the Conneau refinement. In case of without fine-tuning, the models apply the CSLS directly on the mappings learned from the adversarial training, that is, no Procrustes solution–based refinement is done after the adversarial training. This set-up allows us to compare our model directly with the model of Conneau et al. (2018), putting the effect of fine-tuning aside.

Table 6 presents the ablation results for En-Es, En-De, and En-It in both directions. The first row (Conneau-18) presents the results of Conneau et al. (2018) that uses adversarial training to map the word embeddings. The next row shows the results of our full model. The subsequent rows incrementally detach one component from our model. For example, - Enc. adv denotes the variant of our model where the target encoder is not trained on the adversarial loss (θ_E𝒳 in Eq. 13); - - Recon excludes the post-cycle reconstruction loss from - Enc. adv, and - - - Cycle excludes the cycle consistency from - - Recon. Thus, - - - Cycle is a variant of our model that uses only adversarial loss to learn the mapping. However, it is important to note that in contrast to Conneau et al. (2018), our mapping is performed at the code space.

As we compare our full model with the model of Conneau et al. (2018) in the without fine-tuning setting, we notice large improvements in all measures across all data sets: 5.1–7.3% in En → Es, 3–6% in Es → En, 3.4–4.3% in En → De, 1–3% in De → En, 3.4–4.3% in En → It, and 0.3–3.7% in It → En. These improvements demonstrate that our model finds a better mapping compared to Conneau et al. (2018). Among the three components, the cycle consistency is the most influential one across all languages. Training the target encoder adversarially also gives a significant boost. The reconstruction has less impact. If we compare the results of - - - Cycle with Conneau-18, we see sizeable gains for En-Es in both directions. This shows the benefits of mapping at the code level.

Now let us turn our attention to the results with fine-tuning. For a fair comparison with respect to the learned mappings, here we only consider the results of our approach that uses the fine-tuning (refinement) procedure of Conneau et al. (2018). Here also we see gains across all data sets for our model, although the gains are not as verbose as before (about 1% on average). However, this is not surprising as it has been shown that iterative fine-tuning with Procrustes solution is a robust method that can recover many errors made in the initial mapping (Conneau et al. 2018). Given a good enough initial mapping, the measures converge nearly to the same point even though the differences were comparatively more substantial initially; for example, notice the scores are very similar for P@5 and P@10 measures after fine-tuning.

Given the initial mappings and the resulting seed dictionary from our adversarial autoencoder model, we further analyze the impact of different refinement methods. In Sections 3.2.1 and 3.2.2, we described two refinement techniques to fine-tune the initial mappings, namely, refinement with Procrustes solution and with symmetric re-weighting. We then proposed a refinement method (Section 3.2.3) that combines these two techniques. In this section, we analyze the effect of each of these techniques and compare them with the refinement process of Artetxe, Labaka, and Agirre (2018b). Tables 7, 8, and 9 present the results of our experiments on the two data sets using the same initial dictionary for each language pair.

As discussed before, the main phenomenon behind the impressive results of Artetxe, Labaka, and Agirre (2018b) is their refinement procedure. After the robust self-learning step, they perform symmetric re-weighting. We investigate the efficacy of this final symmetric re-weighting step. For the languages on the Conneau data set in Table 7, we see that symmetric re-weighting (Artetxe refinement) boosts the results of robust self-learning by 0.1–0.7% for Spanish, German, and Italian. The improvement is more vivid for En from/to Fi, gaining 2.7% and 3.8%, respectively. For Arabic and low-resource languages in Table 8, we see the similar phenomenon—symmetric re-weighting improves the result ranging from 1.2% to 3.9%. On the Dinu-Artetxe data set in Table 9, we see that symmetric re-weighting step (Artetxe refinement) on top of robust self-learning improves the results by 1.0–3.9%.

Now if we look at the results of our refinement methods in Tables 7–9, we see that our combined method outperforms the individual ones in each of the fourteen (14) translation tasks. If only the Procrustes solution is used for refinement on the Conneau data set, the results lag behind the combined method in the range of 0.4–1.7% for the languages in Table 7 and of 1.8–6.6% for Arabic and low-resource language pairs in Table 9. On the Dinu-Artetxe data set, we see similar phenomenon—results for refinement with only the Procrustes solution are inferior to the combined refinement method in the range of 1.3–2.9%.

Interestingly, when we use only symmetric re-weighting for refinement, the results are close to the results of the combined method. On average, the gain for the combined method over the symmetric re-weighting methods is about 0.85% on the Conneau data set. We observe similar trends on the Dinu-Artetxe data set.

The above observations tell us that the symmetric re-weighting is a powerful method for refinement. In this section, we investigate symmetric re-weighting–based refinement further. Recall that this refinement approach works in three steps: (a) embedding whitening, (b) orthogonal mapping, and (c) embedding de-whitening (see Section 3.2.2). Artetxe, Labaka, and Agirre (2018a) show the correspondence between symmetric re-weighting and the regression-based transformation (see Section 2.1), which are equivalent under certain conditions. The regression-based formulation of the problem gives us further opportunities to explore other possible ways to improve the mapping. For example, dimensionality reduction is one that has been shown to be beneficial in the Canonical Correlation Analysis–based approach (Faruqui and Dyer 2014) and in orthogonal transformation (Smith et al. 2017). The idea is to learn mappings after excluding the features that are not indicative (i.e., have low variance).

Tables 10, 11, and 12 show the results of our experiments on the Conneau and Dinu-Artetxe data sets. The first row in each table shows the results for symmetric re-weighting–based refinement and the remaining rows show the results for regression based solutions. We can observe the benefit of using orthogonality constraint on the mapper. For the language pairs on the Conneau data set in Table 10, adding orthogonality constraint improves the results in the range of 2.2–7.1%, and for the language pairs in Table 11 the improvements are much higher, in the range of 5.4–14.4%. For the language pairs on the Dinu-Artetxe data set in Table 12, we also see the benefit of adding orthogonality constraint.

Now, if we compare the refinement results of the OLS solution with orthogonality constraint with the symmetric re-weighting (first row) on the same induced seed dictionary, we see that for the language pairs on the Conneau data set in Table 10, a small improvement is visible. On the contrary, for other language pairs on the Conneau data set (Table 11) and the language pairs on the Dinu-Artetxe data set (Table 12), OLS with orthogonality constraint lags behind by about 2% on average.

However, we do not see any significant contribution from the regularizers in Tables 10, 11, and 12. After investigation we found that the values in W are generally quite small (in the range of −0.3 to 0.3). As a result, regularizers that penalize large weight values do not seem to have a significant contribution.

In this article, we have proposed an adversarial autoencoder framework to learn the crosslingual mapping of monolingual word embeddings of two languages in a completely unsupervised way. In contrast to the existing methods that directly map word embeddings, our method first learns to transform the embeddings into latent code vectors by pretraining an autoencoder.

We apply adversarial training to map the distributions of the source and target code vectors. In our adversarial training, both the mapper and the target encoder are treated as generators that act jointly to fool the language discriminator. To guide the mapping further, we include constraints for cycle consistency and post-cycle reconstruction.

To improve the initial mapping further, we use two iterative refinement methods—Procrustes solution and symmetric re-weighting—sequentially on the seed dictionary induced from the adversarial training. While the Procrustes solution–based refinement operates in the latent code space, symmetric re-weighting works in the original word embedding space.

Through extensive experimentations on six different language pairs comprising high- and low-resource languages from two different data sources, we demonstrate that our adversarial method outperforms the method of Conneau et al. (2018) for all translation tasks in all measures (P@{1, 5, 10}) across all settings (with and without fine-tuning). Comparison with other existing methods also shows that our method learns better mapping. With an extensive ablation study, we further demonstrated that the cycle consistency is the most important component, followed by the adversarial training of target encoder and the post-cycle reconstruction.

From the in-depth analysis of the refinement procedures, we observe the strength of symmetric re-weighting and the significant effect of orthogonality constraint on it. Our refinement approach that combines both Procrustes solution and symmetric re-weighting achieves the best results across almost all of the translation tasks in two data sets.

The work presented in this article leads to several interesting future directions. In the near future, we plan to incorporate knowledge from the similarity space in our adversarial framework. We also want to use word frequency information in our adversarial model. We would also like to explore the fine-tuning process more rigorously, especially for low-resource languages where the initial mapping obtained from the adversarial training is not that good. Another interesting future direction is to extend our framework to solve downstream crosslingual applications such as machine translation, named entity recognition, part-of-speech tagging, and parsing.

To the best of our knowledge, our work in this article is the first to give a comprehensive overview of existing methods for unsupervised word translation, which is one of the most emerging topics in crosslingual representation learning. We therefore hope that the codebase⁵ released with this article will serve as a benchmark that will facilitate other researchers in pushing the state-of-the-art and in applying bilingual word embeddings to their downstream NLP tasks.

Portions of this work have been published in the NAACL-HLT 2019 conference proceeding (Mohiuddin and Joty 2019). However, this article substantially extends the published work in several ways, most notably: (i) we extend our approach by incorporating two types of refinement (fine-tuning) procedures (Section 3.2); (ii) alongside evaluating the performance with Conneau et al. (2018) refinement and Artetxe, Labaka, and Agirre (2018b) refinement, we also present the performance of our adversarial model with our proposed refinement procedure (Section 5.2); and (iii) we analyze the components of different refinement methods and assess the impact of regularization and orthogonality constraint on refinement methods (Section 5.4). Besides these extensions, most of the article is rewritten to adapt to a journal-style publication.

Many thanks to the anonymous reviewers for their insightful comments on the NAACL-HLT 2019 submitted version.