Nurse is Closer to Woman than Surgeon? Mitigating Gender-Biased Proximities in Word Embeddings

Word embeddings are the standard model for semantic and syntactic representations of words. Unfortunately, these models have been shown to exhibit undesirable word associations resulting from gender, racial, and religious biases. Existing post-processing methods for debiasing word embeddings are unable to mitigate gender bias hidden in the spatial arrangement of word vectors. In this paper, we propose RAN-Debias, a novel gender debiasing methodology that not only eliminates the bias present in a word vector but also alters the spatial distribution of its neighboring vectors, achieving a bias-free setting while maintaining minimal semantic offset. We also propose a new bias evaluation metric, Gender-based Illicit Proximity Estimate (GIPE), which measures the extent of undue proximity in word vectors resulting from the presence of gender-based predilections. Experiments based on a suite of evaluation metrics show that RAN-Debias significantly outperforms the state-of-the-art in reducing proximity bias (GIPE) by at least 42.02%. It also reduces direct bias, adding minimal semantic disturbance, and achieves the best performance in a downstream application task (coreference resolution).


Introduction
Word embedding methods (Devlin et al., 2019;Mikolov et al., 2013a;Pennington et al., 2014) have been staggeringly successful in mapping the semantic space of words to a space of real-valued vectors, capturing both semantic and syntactic relationships. However, as recent research has shown, word embeddings also possess a spectrum of biases related to gender (Bolukbasi et al., 2016;Hoyle et al., 2019), race, and religion (Manzini * Authors have contributed equally.

Word
Neighbours nurse mother 12 , woman 24 , filipina 31 receptionist housekeeper 9 , hairdresser 15 , prostitute 69 prostitute housekeeper 19 , hairdresser 41 , babysitter 44 schoolteacher homemaker 2 , housewife 4 , waitress 8 Table 1: Words and their neighbours extracted using GloVe (Pennington et al., 2014). Subscript indicates the rank of the neighbour. et al., Otterbacher et al., 2017). Bolukbasi et al. (2016) showed that there is a disparity in the association of professions with gender. For instance, while women are associated more closely with 'receptionist' and 'nurse', men are associated more closely with 'doctor' and 'engineer'. Similarly, a word embedding model trained on data from a popular social media platform generates analogies such as "Muslim is to terrorist as Christian is to civilian" (Manzini et al., 2019). Therefore, given the large scale use of word embeddings, it becomes cardinal to remove the manifestation of biases. In this work, we focus on mitigating gender bias from pre-trained word embeddings.
As shown in Table 1, the high degree of similarity between gender-biased words largely results from their individual proclivity towards a particular notion (gender in this case) rather than from empirical utility; we refer to such proximities as "illicit proximities". Existing debiasing methods (Bolukbasi et al., 2016;Kaneko and Bollegala, 2019) are primarily concerned with debiasing a word vector by minimising its projection on the gender direction. Although they successfully mitigate direct bias for a word, they tend to ignore the relationship between a gender-neutral word vector and its neighbours, thus failing to remove the gender bias encoded as illicit proximities between words (Gonen and Goldberg, 2019;Williams et al., 2019). For the sake of brevity, we refer to 'gender-based illicit proximities' as 'illicit proximities' in the rest of the paper.
To account for these problems, we propose a post-processing based debiasing scheme for noncontextual word embeddings, called RAN-Debias (Repulsion, Attraction, and Neutralization based Debiasing). RAN-Debias not only minimizes the projection of gender-biased word vectors on the gender direction but also reduces the semantic similarity with neighbouring word vectors having illicit proximities. We also propose KBC (Knowledge Based Classifier), a word classification algorithm for selecting the set of words to be debiased. KBC utilizes a set of existing lexical knowledge bases to maximize classification accuracy. Additionally, we propose a metric -Genderbased Illicit Proximity Estimate (GIPE), which quantifies gender bias in the embedding space resulting from the presence of illicit proximities between word vectors.
We evaluate debiasing efficacy on various evaluation metrics. For the gender relational analogy test on the SemBias dataset (Zhao et al., 2018b), RAN-GloVe (RAN-Debias applied to GloVe word embedding) outperforms the next best baseline GN-GloVe (debiasing method proposed by Zhao et al. (2018b)) by 21.4% in gender-stereotype type. RAN-Debias also outperforms the best baseline by at least 42.02% in terms of GIPE. Furthermore, the performance of RAN-GloVe on word similarity and analogy tasks on a number of benchmark datasets indicates the addition of minimal semantic disturbance. In short, our major contributions 1 can be summarized as follows: • We provide a knowledge based method (KBC) for classifying words to be debiased.
• We introduce RAN-Debias, a novel approach to reduce both direct and gender-based proximity biases in word embeddings.
• We propose GIPE, a novel metric to measure the extent of undue proximities in word embeddings.
2 Related Work 2.1 Gender Bias in Word Embedding Models Caliskan et al. (2017) highlighted that humanlike semantic biases are reflected through word embeddings (such as GloVe (Pennington et al., 2014)) of ordinary language. They also introduced the Word Embedding Association Test (WEAT) for measuring bias in word embeddings. The 1 The code and data are released at https://github. com/TimeTraveller-San/RAN-Debias authors showed a strong presence of biases in pre-trained word vectors. In addition to gender, they also identified bias related to race. For instance, European-American names are more associated with pleasant terms as compared to African-American names.
In the following subsections, we discuss existing gender debiasing methods based on their mode of operation. Methods that operate on pre-trained word embeddings are known as postprocessing methods, while those which aim to retrain word embeddings by either introducing corpus-level changes or modifying the training objective are known as learning-based methods.
2.2 Debiasing Methods (Post-processing) Bolukbasi et al. (2016) extensively studied gender bias in word embeddings and proposed two debiasing strategies -'hard debias' and 'soft debias'. Hard debias algorithm first determines the direction which captures the gender information in the word embedding space using the difference vectors (e.g., he − she). It then transforms each word vector w to be debiased such that it becomes perpendicular to the gender direction (neutralization). Further, for a given set of word pairs (equalization set), it modifies each pair such that w becomes equidistant to each word in the pair (equalization). On the other hand, the soft debias algorithm applies a linear transformation to word vectors, which preserves pairwise inner products amongst all the word vectors while limiting the projection of gender-neutral words on the gender direction. The authors showed that the former performs better for debiasing than the latter. However, to determine the set of words for debiasing, a support vector machine (SVM) classifier is used, which is trained on a small set of seed words. This makes the accuracy of the approach highly dependent on the generalization of the classifier to all remaining words in the vocabulary. Kaneko and Bollegala (2019) proposed a postprocessing step in which the given vocabulary is split into four classes -non-discriminative femalebiased words (e.g., 'bikini', 'lipstick'), nondiscriminative male-biased words (e.g., 'beard', 'moustache'), gender-neutral words (e.g., 'meal', 'memory'), and stereotypical words (e.g., 'stance librarian', 'doctor'). A set of seed words is then used for each of the categories to train an embedding using an encoder in a denoising au-toencoder, such that gender-related biases from stereotypical words are removed, while preserving feminine information for non-discriminative female-biased words, masculine information for non-discriminative male-biased words, and neutrality of the gender-neutral words. The use of the correct set of seed words is critical for the approach. Moreover, inappropriate associations between words (such as 'nurse' and 'receptionist') may persist. Gonen and Goldberg (2019) showed that current approaches (Bolukbasi et al., 2016;Zhao et al., 2018b), which depend on gender direction for the definition of gender bias and directly target it for the mitigation process, end up hiding the bias rather than reduce it. The relative spatial distribution of word vectors before and after debiasing is similar, and bias-related information can still be recovered. Ethayarajh et al. (2019) provided theoretical proof for hard debias (Bolukbasi et al., 2016) and discussed the theoretical flaws in WEAT by showing that it systematically overestimates gender bias in word embeddings. The authors presented an alternate gender bias measure, called RIPA (Relational Inner Product Association), that quantifies gender bias using gender direction. Further, they illustrated that vocabulary selection for gender debiasing is as crucial as the debiasing procedure. Zhou et al. (2019) investigated the presence of gender bias in bilingual word embeddings and languages which have grammatical gender (such as Spanish and French). Further, they defined semantic gender direction and grammatical gender direction used for quantifying and mitigating gender bias. In this paper, we only focus on languages that have non-gendered grammar (e.g., English). Our method can be applied to any such language.

Debiasing Methods (Learning-based)
Zhao et al. (2018b) developed a word vector training approach, called Gender-Neutral Global Vectors (GN-GloVe) based on the modification of GloVe. They proposed a modified objective function that aims to confine gender-related information to a sub-vector. During the optimization process, the objective function of GloVe is minimized while simultaneously, the square of Euclidean distance between the gender-related sub-vectors is maximized. Further, it is emphasized that the representation of gender-neutral words is perpendic-ular to the gender direction. Being a retraining approach, this method cannot be used on pre-trained word embeddings. Lu et al. (2018) proposed a counterfactual data augmentation (CDA) approach to show that gender bias in language modeling and coreference resolution can be mitigated through balancing the corpus by exchanging gender pairs like 'she ' and 'he' or 'mother' and 'father'. Similarly, Hall Maudslay et al. (2019) proposed a learningbased approach with two enhancements to CDA -a counterfactual data substitution method which makes substitutions with a probability of 0.5 and a method for processing first names based upon bipartite graph matching.
Bordia and Bowman (2019) proposed a genderbias reduction method for word-level language models. They introduced a regularization term that penalizes the projection of word embeddings on the gender direction. Further, they proposed metrics to measure bias at embedding and corpus level. Their study revealed considerable gender bias in Penn Treebank (Marcus et al., 1993) and WikiText-2 (Merity et al., 2018). Mrkšić et al. (2017) defined semantic specialization as the process of refining word vectors to improve the semantic content. Similar to the debiasing procedures, semantic specialization procedures can also be divided into post-processing (Ono et al., 2015;Faruqui and Dyer, 2014) and learning-based (Rothe and Schütze, 2015;Mrkšić et al., 2016;Nguyen et al., 2016) approaches. The performance of post-processing based approaches is shown to be better than learning-based approaches (Mrkšić et al., 2017).

Word Embeddings Specialization
Similar to the 'repulsion' and 'attraction' terminologies used in RAN-Debias, Mrkšić et al. (2017) defined ATTRACT-REPEL algorithm, a post-processing semantic specialization process which uses antonymy and synonymy constraints drawn from lexical resources. Although it is superficially similar to RAN-Debias, there are a number of differences between the two approaches. Firstly, the ATTRACT-REPEL algorithm operates over mini-batches of synonym and antonym pairs, while RAN-Debias operates on a set containing gender-neutral and gender-biased words. Secondly, the 'attract' and 'repel' terms carry different meanings with respect to the algo-

Notation Denotation
w Vector corresponding to a word w w d Debiased version of w V Vocabulary set V p The set of words which are preserved during the debiasing procedure V d The set of words which are subjected to the debiasing procedure D Set of dictionaries d i A particular dictionary from the set D g Gender direction Direct bias of a word w. β( w 1 , w 2 ) Indirect bias between a pair of words w 1 and w 2 . η( w) Gender-based proximity bias of a word w N w Set of neighbouring words of a word w F r ( w d ) Repulsion objective function Gender-based Illicit Proximity Estimate

Proposed Approach
Given a set of pre-trained word vectors over a vocabulary set V, we aim to create a trans- such that the stereotypical gender information present in the resulting embedding set are minimized with minimal semantic offset. We first define the categories into which each word w ∈ V is classified in a mutually exclusive manner. Table 2 summarizes important notations used throughout the paper.
• Preserve set (V p ): This set consists of words for which gender carries semantic importance; such as names, gendered pronouns and words like 'beard' and 'bikini' which have a meaning closely associated with gender. In addition, words which are non-alphabetic are also included as debiasing them will be of no practical utility.
• Debias set (V d ): This set consists of all the words in the vocabulary which are not present in V p . These words are expected to be gender-neutral in nature and hence subjected to debiasing procedure. Note that V d not only consists of genderstereotypical words (e.g., 'nurse', 'warrior', 'receptionist', etc.), but also gender-neutral words (e.g., 'sky', 'table', 'keyboard', etc.).

Word Classification Methodology
Prior to the explanation of our method, we present the limitations of previous approaches for word classification. Bolukbasi et al. (2016) trained a linear SVM using a set of gender-specific seed words, which is then generalized on the whole embedding set to identify other gender-specific words. However, such methods rely on training a supervised classifier on word vectors, which are themselves gender-biased. Because such classifiers are trained on biased data, they catch onto the underlying gender-bias cues and often misclassify words. For instance, the SVM classifier trained by Bolukbasi et al. (2016) misclassifies the word 'blondes' as gender-specific, among others. Further, we empirically show the inability of a supervised classifier (SVM) to generalize over the whole embedding using various metrics in Table 3.
Taking into consideration this limitation, we propose Knowledge Based Classifier (KBC) that relies on knowledge bases instead of word embeddings, thereby circumventing the addition of bias in the classification procedure. Moreover, unlike RIPA (Ethayarajh et al., 2019), our approach does not rely on creating a biased direction that may be difficult to determine. Essentially, KBC relies on the following assumption.
Assumption 1. If there exists a dictionary d such that it stores a definition d[w] corresponding to a word w, then w can be defined as genderspecific or not based on the existence or absence of a gender-specific reference s ∈ seed in the definition d [w], where the set seed consists of gender-specific references such as {'man', 'woman', 'boy', 'girl'}.
Algorithm 1 formally explains KBC. We denote Algorithm 1: Knowledge Based Classifier (KBC) Input : V : vocabulary set, isnonaphabetic(w): checks for non-alphabetic words seed: set of gender-specific words stw: set of stop words names: set of gender-specific names represents the definition of a word w. Output: V p : set of words that will be preserved, V d : set of words that will be debiased each if condition as a stage and explain it below: • Stage 1: This stage classifies all stop words and non-alphabetic words as V p . Debiasing such words serve no practical utility; hence we preserve them.
• Stage 2: This stage classifies all words that belong to either names set or seed set as V p . Set names is collected from open source knowledge base 2 . Set seed consists of gender-specific reference terms. We preserve names, as they hold important gender information (Pilcher, 2017).
• Stage 3: This stage uses a collection of dictionaries to determine if a word is gender-specific using Assumption 1. To counter the effect of biased definition arising from any particular dictionary, we make a decision based upon the consensus of all dictionaries. A word is classified as gender-specific and added to V p if and only if more than half of the dictionaries classify it as gender-specific. In our experiments, we employ WordNet (Miller, 1995) and the Oxford dictionary. As pointed out by Bolukbasi et al. (2016), Word-Net consists of few definitions which are gender-2 https://github.com/ganoninc/ fb-gender-json biased such as the definition of 'vest'; therefore, by utilizing our approach, we counter such cases as the final decision is based upon consensus.
The remaining words that are not preserved by KBC are categorized into V d . It is the set of words which are debiased by RAN-Debias later.

Types of Gender Bias
First, we briefly explain two types of gender bias as defined by Bolukbasi et al. (2016) and then introduce a new type of gender bias resulting from illicit proximities in word embedding space.
• Direct Bias (D b ): For a word w, the direct bias is defined by, where, g is the gender direction measured by taking the first principal component from the principal component analysis of ten gender pair difference vectors, such as ( he − she) as mentioned in (Bolukbasi et al., 2016), and c represents the strictness of measuring bias.
• Indirect Bias (β): The indirect bias between a given pair of words w and v is defined by, Here, w and v are normalized. w ⊥ is orthogonal to the gender direction g: w ⊥ = w − w g , and w g is the contribution from gender: w g = ( w. g) g. Indirect bias measures the change in the inner product of two word vectors as a proportion of the earlier inner product after projecting out the gender direction from both the vectors. A higher indirect bias between two words indicates a strong association due to gender.
• Gender-based Proximity Bias (η): Gonen and Goldberg (2019) observed that the existing debiasing methods are unable to completely debias word embeddings because the relative spatial distribution of word embeddings after the debiasing process still encapsulates bias-related information. Therefore, we propose gender-based proximity bias that aims to capture the illicit proximities arising between a word and its closest k neighbours due to gender-based constructs. For a given word w i ∈ V d , the gender-based proximity bias η w i is defined as: where, and θ s is a threshold for indirect bias.
The intuition behind this is as follows. The set N w i consists of the top k neighbours of w i calculated by finding the word vectors having the maximum cosine similarity with w i . Further, N b w i ⊆ N w i is the set of neighbours having indirect bias β greater than a threshold θ s , which is a hyperparameter that controls neighbour deselection on the basis of indirect bias. The lower is the value of θ s , the higher is the cardinality of indicates that the neighbourhood of the word is gender-biased.

Proposed Method -RAN-Debias
We propose a multi-objective optimization based solution to mitigate both direct 3 and gender-based proximity bias while adding minimal impact to the semantic and analogical properties of the word embedding. For each word w ∈ V d and its vector w ∈ R h , where h is the embedding dimension, we find its debiased counterpart w d ∈ R h by solving the following multi-objective optimization problem: We solve this by formulating a single objective F ( w d ) and scalarizing the set of objectives using the weighted sum method as follows: is minimized using the adam (Kingma and Ba, 2015) optimized gradient descent to obtain the optimal debiased embedding w d . As shown in the subsequent sections, the range of objective functions F r , F a , F n (defined later) is [0, 1]; thus we use the weights λ i for determining the relative importance of one objective function over another.

Repulsion
For any word w ∈ V d , we aim to minimize the gender bias based illicit associations. Therefore, our objective function aims to 'repel' w d from the neighbouring word vectors which have a high value of indirect bias (β) with it. Consequently, we name it 'repulsion' (F r ) and primarily define the repulsion set S r to be used in F r as follows.
Definition 1. For a given word w, the repulsion set S r is defined as S r = {n i : n i ∈ N w and β( w, n i ) > θ r }, where N w is the set of top 100 neighbours obtained from the original word vector w.
Since we aim to reduce the unwanted semantic similarity between w d and the set of vectors S r , we define the objective function F r as follows.
For our experiments, we find that θ r = 0.05 is the appropriate threshold to repel majority of genderbiased neighbours.

Attraction
For any word w ∈ V d , we aim to minimize the loss of semantic and analogical properties for its debiased counterpart w d . Therefore, our objective function aims to attract w d towards w in the word embedding space. Consequently, we name it 'attraction' (F a ) and define it as follows:

Neutralization
For any word w ∈ V d , we aim to minimize its bias towards any particular gender. Therefore, the objective function F n represents the absolute value of dot product of word vector w d with the gender direction g (as defined by Bolukbasi et al. (2016)). Consequently, we name it 'neutralization' (F n ) and define it as follows:

Time complexity of RAN-Debias
Computationally, there are two major components of RAN-Debias: 1. Calculate neighbours for each word w ∈ V d and store them in a hash table. This has a time complexity of O(n 2 ) where n = |V d |. We illustrate four possible scenarios with respect to their effect on GIPE, with θ s = 0.05: (b) An edge with β < θ s may not contribute to γ i or η w i ; (c) An outgoing edge may contribute to η w i only; (d) An incoming edge may contribute to γ i only; (e) Incoming and outgoing edges may contribute to γ i and η w i respectively. Every node pair association can be categorized as one of the aforementioned four cases.
2. Debias each word using gradient descent, whose time complexity is O(n). The overall complexity of RAN-Debias is O(n 2 ) i.e., quadratic with respect to the cardinality of debias set V d .

Gender-based Illicit Proximity Estimate (GIPE)
In Section 3.2, we defined the gender proximity bias (η). In this section, we extend it to the embedding level for generating a global estimate. Intuitively, an estimate can be generated by simply taking the mean of η w , ∀w ∈ V d . However, this computation assigns equal importance to all η w values, which is an oversimplification. A word w may itself be in the proximity of another word w ∈ V d through gender-biased associations, thereby increasing η w . Such cases in which w increases η w for other words should also be taken into account. Therefore, we use a weighted average of η w , ∀w ∈ V for determining a global estimate. We first define a weighted directed network, called Bias Based Network (BBN). The use of a graph data structure makes it easier to understand the intuition behind GIPE.
Definition 2. Given a set of non gender-specific words W , bias based network is a directed graph G = (V, E), where nodes represent word vectors and weights of directed edges represent the indirect bias (β) between them. The vertex set V and edge set E are obtained according to Algorithm 2.
Algorithm 2: Compute BBN for the given set of word vectors Input : ξ: word embedding set, W : set of non gender-specific words, n: number of neighbours Output: G: bias based network For each word w i in W , we find N , the set of top n word vectors having the highest cosine similarity with w i (we keep n to be 100 to reduce computational overhead without comprising on qual-ity). For each pair ( w i , w k ), where w k ∈ N , a directed edge is assigned from w i to w k with the edge weight being β( w i , w k ). In case the given embedding is a debiased version, we use the nondebiased version of the embedding for computing β( w i , w k ). Figure 1a portrays a sub-graph in BBN. By representing the set of non genderspecific words as a weighted directed graph we can use the number of outgoing and incoming edges for a node (word w i ) for determining η w i and its weight respectively, thereby leading to the formalization of GIPE as follows.
Definition 3. For a BBN G, the Gender-based Illicit Proximity Estimate of G, indicated by GIP E(G) is defined as: where, for a word w i , η w i is the gender-based proximity bias as defined earlier, is a (small) positive constant, and γ i is the weight, defined as: The intuition behind the metric is as follows. For a bias based network G, GIP E(G) is the weighted average of gender-based proximity bias (η w i ) for all nodes w i ∈ W , where the weight of a node is γ i , which signifies the importance of the node in contributing towards the genderbased proximity bias of other word vectors. γ i takes into account the number of incoming edges having β higher than a threshold θ s . Therefore, we take into account how the neighbourhood of a node contributes towards illicit proximities (having high β values for outgoing edges) as well as how a node itself contributes towards illicit proximities of other nodes (having high β values for incoming edges). For illustration, we analyze a subgraph in Figure 1. By incorporating γ i , we take into account both dual ( Figure 1e) and incoming (Figure 1d) edges, which would not have been the case otherwise. In GloVe (2017-January dump of Wikipedia), the word 'sweetheart' has 'nurse' in the set of its top 100 neighbours and β > θ s ; however, 'nurse' does not have 'sweetheart' in the set of its top 100 neighbours. Hence, while 'nurse' contributes towards gender-based proximity bias of the word 'sweetheart', vice versa is not true. Similarly, if dual-edge exists, then both γ i and η w i are taken into account. Therefore, GIPE considers all possible cases of edges in BBN, making it a holistic metric.

Experiment Results
We conduct the following performance evaluation tests: • We compare KBC with SVM based (Bolukbasi et al., 2016) and RIPA based (Ethayarajh et al., 2019) methods for word classification.
• We evaluate the capacity of RAN-Debias on GloVe (aka RAN-GloVe) for the gender relational analogy dataset -SemBias (Zhao et al., 2018b).
• We demonstrate the ability of RAN-GloVe to mitigate gender proximity bias by computing and contrasting the GIPE value.
• We evaluate RAN-GloVe on several benchmark datasets for similarity and analogy tasks, showing that RAN-GloVe introduces minimal semantic offset to ensure quality of the word embeddings.
• We demonstrate that RAN-GloVe successfully mitigates gender bias in a downstream application -coreference resolution.
Although we report and analyze the performance of RAN-GloVe in our experiments, we also applied RAN-Debias to other popular noncontextual and monolingual word embedding, Word2vec (Mikolov et al., 2013a) to create RAN-Word2vec. As expected, we observed similar results (hence not reported for the sake of brevity), emphasizing the generality of RAN-Debias. Note that the percentages mentioned in the rest of the section are relative unless stated otherwise.

Training Data and Weights
We use GloVe (Pennington et al., 2014) trained on 2017-January dump of Wikipedia, consisting of 322, 636 unique word vectors of 300 dimensions. We apply KBC on the vocabulary set V obtaining V p and V d of size 47,912 and 274,724 respectively. Further, judging upon the basis of performance evaluation tests as discussed above, we experimentally select the weights in Equation 3 as λ 1 = 1/8, λ 2 = 6/8, and λ 3 = 1/8.

Baselines for Comparisons
We compare RAN-GloVe against the following word embedding models, each of which is trained on the 2017-January dump of Wikipedia. • GN-GloVe: Gender-neutral GloVe; we use the original 5 debiased version of GloVe released by Zhao et al. (2018b).
• GP-GloVe: Gender-preserving GloVe; we use the original 6 debiased version of GloVe released by Kaneko and Bollegala (2019).

Word Classification
We compare KBC with RIPA based (unsupervised) (Ethayarajh et al., 2019) and SVM based (supervised) (Bolukbasi et al., 2016) approaches for word classification. We create a balanced labeled test set consisting of a total of 704 words, with 352 words for each category -genderspecific and non gender-specific. For the non gender-specific category, we select all the 87 neutral and biased words from the SemBias dataset (Zhao et al., 2018b). Further, we select all 320, 40 and 60 gender-biased occupation words released by Bolukbasi et al. (2016); Zhao et al. (2018a) and Rudinger et al. (2018) respectively. After combining and removing duplicate words from them, we obtain 352 non gender-specific words. For the gender-specific category, we use a list of 222 male and 222 female words provided by Zhao et al. (2018b). We use stratified sampling to undersample 444 words into 352 words for balancing the classes. The purpose of creating this diversely sourced dataset is to provide a robust ground-truth for evaluating the efficacy of different word classification algorithms. Table 3 shows precision, recall, F1-score, AUC-ROC, and accuracy by considering gender-specific words as the positive class and non gender-specific words as the negative class. Thus, for KBC, we consider the output set V p as the positive and V d as the negative class.
The SVM based approach achieves high precision but at the cost of a low recall. While the majority of the words classified as gender-specific are correct, it achieves this due to the limited coverage of the rest of gender-specific words, resulting in them being classified as non gender-specific, thereby reducing the recall drastically.   The RIPA approach performs fairly with respect to precision and recall. Unlike SVM, RIPA is not biased towards a particular class and results in rather fair performance for both the classes. Almost similar to SVM, KBC also correctly classifies most of the gender-specific words but in an exhaustive manner, thereby leading to much fewer misclassification of gender-specific words as non gender-specific. As a result, KBC achieves sufficiently high recall.
Overall, KBC outperforms the best baseline by an improvement of 2.7% in AUC-ROC, 15.6% in F1-score, and 9.0% in the accuracy. Additionally, since KBC entirely depends on knowledge bases, the absence of a particular word in them may result in misclassification. This could be the reason behind the lower precision of KBC as compared to SVM-based classification and can be improved upon by incorporating more extensive knowledge bases.

Gender Relational Analogy
To evaluate the extent of gender bias in RAN-GloVe, we perform gender relational analogy test on the SemBias (Zhao et al., 2018b) dataset. Each instance of SemBias contains four types of word pairs: a gender-definition word pair (Definition; 'headmaster-headmistress'), a gender-stereotype word pair (Stereotype; 'manager-secretary') and two other word pairs which have similar meanings but no gender-based relation (None; 'treblebass'). There are a total of 440 instances in the  semBias dataset, created by the cartesian product of 20 gender-stereotype word pairs and 22 gender-definition word pairs. From each instance, we select a word pair (a, b) from the four word pairs such that using the word embeddings under evaluation, cosine similarity of the word vectors ( he − she) and ( a − b) would be maximum. Table 4 shows an embedding-wise comparison on the SemBias dataset. The accuracy is measured in terms of the percentage of times each type of word pair is selected as the top for various instances. RAN-GloVe outperforms all other post-processing debiasing methods by achieving at least 9.96% and 82.8% better accuracy in gender-definition and gender-stereotype, respectively. We attribute this performance to be an effect of superior vocabulary selection by KBC and the neutralization objective of RAN-Debias. KBC classifies the words to be debiased or preserved with high accuracy, while the neutralization objective function of RAN-Debias directly minimizes the preference of a biased word between 'he' and 'she'; reducing the gender cues that give rise to unwanted genderbiased analogies (Table 10). Therefore, although RAN-GloVe achieves lower accuracy for genderdefinition type as compared to (learning-based) GN-GloVe, it outperforms the next best baseline in Stereotype by at least 21.4%.

Gender-based Illicit Proximity Estimate
GIPE analyses the extent of undue gender bias based proximity between word vectors. An embedding-wise comparison for various values of θ s is presented in Table 5. For a fair comparison, we compute GIPE for a BBN created upon our debias set V d as well as for H d , the set of words debiased by Bolukbasi et al. (2016).
Here, θ s represents the threshold as defined ear-lier in Equation 4. As it may be inferred from Equations 1 and 4, upon increasing the value of θ s , for a word w i , the value of both η w i and γ i decreases, as lesser number of words qualify the threshold for selection in each case. Therefore, as evident from Table 5, value of GIPE decreases with the increase of θ s . For the input set V d , RAN-GloVe outperforms the next best baseline (Hard-GloVe) by at least 42.02%. We attribute this to the inclusion of the repulsion objective function F r in Equation 2, which reduces the unwanted gender-biased associations between words and their neighbours. For the input set H d , RAN-GloVe performs better than other baselines for all values of θ s except for θ s = 0.07 where it closely follows Hard-GloVe.
Additionally, H d consists of many misclassified gender-specific words, as observed from the low recall performance at the word classification test in Section 4.3. Therefore, the values of GIPE corresponding to every value of θ s for the input H d is higher as compared to the values for V d .
Although there is a significant reduction in GIPE value for RAN-GloVe as compared to other word embedding models, word pairs with noticeable β values still exist (as indicated by non-zero GIPE values), which is due to the trade-off between semantic offset and bias reduction. As a result, GIPE for RAN-GloVe is not perfectly zero but close to it.

Analogy Test
The task of analogy test is to answer the following question: "p is to q as r is to ?". Mathematically, it aims at finding a word vector w s which has the maximum cosine similarity with ( w q − w p + w r ). However, Schluter (2018) highlights some critical issues with word analogy tests. For instance, there is a mismatch between the distributional hypothesis used for generating word vectors and the word analogy hypothesis. Nevertheless, following the practice of using word analogy test to ascertain the semantic prowess of word vectors, we evaluate RAN-GloVe to provide a fair comparison with other baselines.
We use Google (Mikolov et al., 2013a) (semantic (Sem) and syntactic (Syn) analogies, containing a total 19,556 questions) and MSR (Mikolov et al., 2013b) (containing a total 7,999 syntactic questions) datasets for evaluating the performance of word embeddings. We use 3COSMUL(Levy  Table 6: Comparison of various embedding methods for (a) analogy tests (performance is measured in accuracy) and (b) word semantic similarity tests (performance is measured in terms of Spearman rank correlation).
and Goldberg, 2014) for finding w s .
Table 6(a) shows that RAN-GloVe outperforms other baselines on the Google (Sem and Syn) dataset while closely following on the MSR dataset. The improvement in performance can be attributed to the removal of unwanted neighbours of a word vector (having gender bias based proximity), while enriching the neighbourhood with those having empirical utility, leading to a better performance in analogy tests.

Word Semantic Similarity Test
Word semantic similarity task is a measure of how closely a word embedding model captures the similarity between two words as compared to humanannotated ratings. For a word pair, we compute the cosine similarity between the word embeddings and its Spearman correlation with the human ratings. The word pairs are selected from the following benchmark datasets: RG (Rubenstein and Goodenough, 1965), MTurk (Radinsky et al., 2011), RW (Luong et al., 2013, MEN (Bruni et al., 2014), SimLex999 (Hill et al., 2015) and AP (Almuhareb and Poesio, 2005). The results for these tests are obtained from the word embedding benchmark package (Jastrzebski et al., 2017) 7 . Note that it is not our primary aim to achieve a state-of-the-art result in this test. It is only considered to evaluate semantic loss. Table 6(b) shows that RAN-GloVe performs better or follows closely to the best baseline. This shows that RAN-Debias introduces minimal semantic disturbance.

Coreference Resolution
Finally, we evaluate the performance of RAN-GloVe on a downstream application task -coreference resolution. The aim of coreference resword-embeddings-benchmarks olution is to identify all expressions which refer to the same entity in a given text. We evaluate the embedding models on the OntoNotes 5.0 (Weischedel et al., 2012) and the WinoBias (Zhao et al., 2018a) benchmark datasets. WinoBias comprises of sentences constrained by two prototypical templates (Type 1 and Type 2), where each template is further divided into two subsets (PRO and ANTI). Such a construction facilitates in revealing the extent of gender bias present in coreference resolution models. While both templates are designed to assess the efficacy of coreference resolution models, Type 1 is exceedingly challenging as compared to Type 2 as it has no syntactic cues for disambiguation. Each template consists of two subsets for evaluation -pro-stereotype (PRO) and anti-stereotype (ANTI). PRO consists of sentences in which the gendered pronouns refer to occupations biased towards the same gender. For instance, consider the sentence "The doctor called the nurse because he wanted a vaccine." Stereotypically, 'doctor' is considered to be a male-dominated profession, and the gender of pronoun referencing it ('he') is also male. Therefore, sentences in PRO are consistent with societal stereotypes. ANTI consists of the same sentences as PRO, but the gender of the pronoun is changed. Considering the same example but by replacing 'he' with 'she', we get: "The doctor called the nurse because she wanted a vaccine." In this case, the gender of pronoun ('she') which refers to 'doctor' is female. Therefore, sentences in ANTI are not consistent with societal stereotypes. Due to such construction, gender bias in the word embeddings used for training the coreference model would naturally perform better in PRO than ANTI and lead to a higher absolute difference (Diff) between them. While a lesser gender bias in the model would attain a smaller Diff, the ideal case produces an absolute difference of zero.  Table 7: F1-Score (in %) in the task of coreference resolution. Diff denotes the absolute difference between F1-score on PRO and ANTI datasets.
Following the coreference resolution testing methodology used by Zhao et al. (2018b), we train the coreference resolution model proposed by Lee et al. (2017) on the OntoNotes train dataset for different embeddings. Table 7 shows F1score on OntoNotes 5.0 test set, WinoBias PRO and ANTI test set for Type 1 template, along with the absolute difference (Diff) of F1-scores on PRO and ANTI datasets for different word embeddings. The results for GloVe, Hard-GloVe, and GN-GloVe are obtained from Zhao et al. (2018b). Table 7 shows that RAN-GloVe achieves the smallest absolute difference between scores on PRO and ANTI subsets of WinoBias, significantly outperforming other embedding models and achieving 97.4% better Diff (see Table 7 for the definition of Diff) than the next best baseline (Hard-GloVe) and 98.7% better than the original GloVe. This lower Diff is achieved by an improved accuracy in ANTI and a reduced accuracy in PRO. We hypothesise that the high performance of non-debiased GloVe in PRO is due to the unwanted gender cues rather than the desired coreference resolving ability of the model. Further, the performance reduction in PRO for the other debiased versions of GloVe also corroborates this hypothesis. Despite debiasing GloVe, a considerable amount of gender cues remain in the baseline models as quantified by a lower, yet significant Diff. In contrast, RAN-GloVe is able to remove gender cues dramatically, thereby achieving an almost ideal Diff. Additionally, the performance of RAN-GloVe on the OntoNotes 5.0 test set is comparable with that of other embeddings.

Ablation Study
To quantitatively and qualitatively analyze the effect of neutralization and repulsion in RAN-Debias, we perform an ablation study. We examine the following changes in RAN-Debias independently: 1. Nullify the effect of repulsion by setting λ 1 = 0, thus creating AN-GloVe. 2. Nullify the effect of neutralization by setting λ 3 = 0, thus creating RA-GloVe. We demonstrate the effect of the absence of neutralization or repulsion through a comparative analysis on GIPE and the SemBias analogy test.  The GIPE values for AN-GloVe, RA-GloVe, and RAN-GloVe are presented in Table 8. We observe that in the absence of repulsion (AN-GloVe), the performance is degraded by at least 72% compared to RAN-GloVe. It indicates the efficacy of repulsion in our objective function as a way to reduce the unwanted gender-biased associations between words and their neighbours, thereby reducing GIPE. Further, even in the absence of neutralization (RA-GloVe), GIPE is worse by at least 50% as compared to RAN-GloVe. In fact, the minimum GIPE is observed for RAN-GloVe, where both repulsion and neutralization are used in synergy as compared to the absence of any one of them.
To illustrate further, Table 9 shows the rank of neighbours having illicit proximities for three professions, using different version of debiased embeddings. It can be observed that the ranks in RA-GloVe are either close to or further away from the ranks in AN-GloVe, highlighting the importance of repulsion in the objective function. Further, the ranks in RAN-GloVe are the farthest, corroborating the minimum value of GIPE as observed in Table 8. Table 10 shows that in the absence of neutralization (RA-GloVe), the tendency of favouring stereotypical analogies increases by an absolute difference of 6.2% as compared to RAN-GloVe. On the other hand, through the presence of neutralization, AN-GloVe does not favour stereotypical analogies. This suggests that reducing the projection of biased words on gender direction through neutralization is an effective measure to reduce stereotypical analogies within the embedding space. For example, consider the following Word Neighbour Embedding AN-GloVe RA-GloVe RAN- GloVe   Captain  sir  28  22  52  james  26  30  75   Nurse  women  57  56  97  mother  49  74  144   Farmer  father  22  54  86  son  45  90  162   Table 9: For three professions, we compare the ranks of their neighbours due to illicit proximities (the values denote the ranks).  During the evaluation, RA-GloVe incorrectly selects the gender-stereotype word pair as the closest analogy with (he, she), while AN-GloVe and RAN-GloVe correctly select the gender-definition word pair. Further, we observe that RAN-GloVe is able to maintain the high performance of AN-GloVe, and the difference is less (0.2% compared to 1.1%) which is compensated by the superior performance of RAN-GloVe over other metrics like GIPE. Through this ablation study, we understand the importance of repulsion and neutralization in the multi-objective optimization function of RAN-Debias. The superior performance of RAN-GloVe can be attributed to the synergistic interplay of repulsion and neutralization. Hence, in RAN-GloVe we attain the best of both worlds.

Case study -Neighbourhood of Words
Here we highlight the changes in the neighbourhood (collection of words sorted in the descending order of cosine similarity with the given word) of words before and after the debiasing process. To maintain readability while also demonstrating the changes in proximity, we only analyze a few selected words. However, our proposed metric GIPE quantifies this for an exhaustive vocabulary set.
We select a set of gender-neutral professions having high values of gender-based proximity bias η w i as defined earlier. For each of these professions, in Table 11, we select a set of four words from their neighbourhood for two classes: • Class A: Neighbours arising due to genderbased illicit proximities.
• Class B: Neighbours whose proximities are not due to any kind of bias.
For the words in class A, the debiasing procedure is expected to increase their rank, thereby decreasing the semantic similarity, while for words belonging to class B, debiasing procedure is expected to retain or improve the rank for maintaining the semantic information.
We observe that RAN-GloVe not only maintains the semantic information by keeping the rank of words in class B close to their initial value but unlike other debiased embeddings, it drastically increases the rank of words belonging to class A. However, in some cases like the word 'Socialite', we observe that the ranks of words such as 'businesswoman' and 'heiress', despite belonging to class A, are close to their initial values. This can be attributed to the high semantic dependence of 'Socialite' on these words, resulting in a bias removal and semantic information trade-off.

Conclusion
In this paper, we proposed a post-processing gender debiasing method, called RAN-Debias. Our method not only mitigates direct bias of a word but also reduces its associations with other words that arise from gender-based predilections. We also proposed a word classification method, called KBC, for identifying the set of words to be debiased. Instead of using 'biased' word embeddings, KBC uses multiple knowledge bases for word classification. Moreover, we proposed Genderbased Illicit Proximity Estimate (GIPE), a metric to quantify the extent of illicit proximities in an embedding. RAN-Debias significantly outperformed other debiasing methods on a suite of evaluation metrics, along with the downstream application task of coreference resolution while introducing minimal semantic disturbance.
In the future, we would like to enhance KBC by utilizing machine learning methods to account for the words which are absent in the knowledge base. Currently, RAN-Debias is directly applicable to non-contextual word embeddings for non-