Bias and Fairness in Large Language Models: A Survey

Warning: This article contains explicit statements of offensive or upsetting language.

The rise and rapid advancement of large language models (LLMs) has fundamentally changed language technologies (e.g., Brown et al. 2020; Conneau et al. 2020; Devlin et al. 2019; Lewis et al. 2020; Liu et al. 2019; OpenAI 2023; Radford et al. 2018, 2019; Raffel et al. 2020). With the ability to generate human-like text, as well as adapt to a wide array of natural language processing (NLP) tasks, the impressive capabilities of these models have initiated a paradigm shift in the development of language models. Instead of training task-specific models on relatively small task-specific datasets, researchers and practitioners can use LLMs as foundation models that can be fine-tuned for particular functions (Bommasani et al. 2021). Even without fine-tuning, foundation models increasingly enable few- or zero-shot capabilities for a wide array of scenarios like classification, question-answering, logical reasoning, fact retrieval, information extraction, and more, with the task described in a natural language prompt to the model and few or no labeled examples (e.g., Brown et al. 2020; Kojima et al. 2022; Liu et al. 2023; Radford et al. 2019; Wei et al. 2022; Zhao et al. 2021).

Lying behind these successes, however, is the potential to perpetuate harm. Typically trained on an enormous scale of uncurated Internet-based data, LLMs inherit stereotypes, misrepresentations, derogatory and exclusionary language, and other denigrating behaviors that disproportionately affect already-vulnerable and marginalized communities (Bender et al. 2021; Dodge et al. 2021; Sheng et al. 2021b). These harms are forms of “social bias,” a subjective and normative term we broadly use to refer to disparate treatment or outcomes between social groups that arise from historical and structural power asymmetries, which we define and discuss in Section 2.¹ Though LLMs often reflect existing biases, they can amplify these biases, too; in either case, the automated reproduction of injustice can reinforce systems of inequity (Benjamin 2020). From negative sentiment and toxicity directed towards some social groups, to stereotypical linguistic associations, to lack of recognition of certain language dialects, the presence of biases of LLMs have been well-documented (e.g., Blodgett and O’Connor 2017; Hutchinson et al. 2020; Mei, Fereidooni, and Caliskan 2023; Měchura 2022; Mozafari, Farahbakhsh, and Crespi 2020; Sap et al. 2019; Sheng et al. 2019).

With the growing recognition of the biases embedded in LLMs has emerged an abundance of works proposing techniques to measure or remove social bias, primarily organized by (1) metrics for bias evaluation, (2) datasets for bias evaluation, and (3) techniques for bias mitigation. In this survey, we categorize, summarize, and discuss each of these areas of research. For each area, we propose an intuitive taxonomy structured around the types of interventions to which a researcher or practitioner has access. Metrics for bias evaluation are organized by the underlying data structure assumed by the metric, which may differ depending on access to the LLM (i.e., can the user access model-assigned token probabilities, or only generated text output?). Datasets are similarly categorized by their structure. Techniques for bias mitigation are organized by the stage of intervention: pre-processing, in-training, intra-processing, and post-processing.

The key contributions of this work are as follows:

1. A consolidation, formalization, and expansion of social bias and fairness definitions for NLP. We disambiguate the types of social harms that may emerge from LLMs, consolidating literature from machine learning, NLP, and (socio)linguistics to define several distinct facets of bias. We organize these harms in a taxonomy of social biases that researchers and practitioners can leverage to describe bias evaluation and mitigation efforts with more precision. We shift fairness frameworks typically applied to machine learning classification problems towards NLP and introduce several fairness desiderata that begin to operationalize various fairness notions for LLMs. We aim to enhance understanding of the range of bias issues, their harms, and their relationships to each other.

2. A survey and taxonomy of metrics for bias evaluation. We characterize the relationship between evaluation metrics and datasets, which are often conflated in the literature, and we categorize and discuss a wide range of metrics that can evaluate bias at different fundamental levels in a model: embedding-based (using vector representations), probability-based (using model-assigned token probabilities), and generated text-based (using text continuations conditioned on a prompt). We formalize metrics mathematically with a unified notation that improves comparison between metrics. We identify limitations of each class of metrics to capture downstream application biases, highlighting areas for future research.

3. A survey and taxonomy of datasets for bias evaluation, with a compilation of publicly available datasets. We categorize several datasets by their data structure: counterfactual inputs (pairs of sentences with perturbed social groups) and prompts (phrases to condition text generation). With this classification, we leverage our taxonomy of metrics to highlight compatibility of datasets with new metrics beyond those originally posed. We increase comparability between dataset contents by identifying the types of harm and the social groups targeted by each dataset. We highlight consistency, reliability, and validity challenges in existing evaluation datasets as areas for improvement. We share publicly available datasets here: https://github.com/i-gallegos/Fair-LLM-Benchmark

4. A survey and taxonomy of techniques for bias mitigation. We classify an extensive range of bias mitigation methods by their intervention stage: pre-processing (modifying model inputs), in-training (modifying the optimization process), intra-processing (modifying inference behavior), and post-processing (modifying model outputs). We construct granular subcategories at each mitigation stage to draw similarities and trends between classes of methods, with mathematical formalization of several techniques with unified notation, and representative examples of each class of method. We draw attention to ways that bias may persist at each mitigation stage.

5. An overview of key open problems and challenges that future work should address. We challenge future research to address power imbalances in LLM development, conceptualize fairness more robustly for NLP, improve bias evaluation principles and standards, expand mitigation efforts, and explore theoretical limits for fairness guarantees.

Each taxonomy provides a reference for researchers and practitioners to identify which metrics, datasets, or mitigations may be appropriate for their use case, to understand the tradeoffs between each technique, and to recognize areas for continued exploration.

This survey complements existing literature by offering a more extensive and comprehensive examination of bias and fairness in NLP. Surveys of bias and fairness in machine learning, such as Mehrabi et al. (2021) and Suresh and Guttag (2021), offer important broad-stroke frameworks, but are not specific to linguistic tasks or contexts. While previous work within NLP such as Czarnowska, Vyas, and Shah (2021), Kumar et al. (2023b), and Meade, Poole-Dayan, and Reddy (2021) has focused on specific axes of bias evaluation and mitigation, such as extrinsic fairness metrics, empirical validation, and language generation interventions, our work provides increased breadth and depth. Specifically, we offer a comprehensive overview of bias evaluation and mitigation techniques across a wide range of NLP tasks and applications, synthesizing diverse bodies of work to surface unifying themes and overarching challenges. Beyond enumerating techniques, we also examine the limitations of each class of approach, providing insights and recommendations for future work.

We do not attempt to survey the abundance of work on algorithmic fairness more generally, or even bias in all language technologies broadly. In contrast, we focus solely on bias issues in LLMs for English (with additional languages for machine translation and multilingual models), and restrict our search to works that propose novel closed-form metrics, datasets, or mitigation techniques; for our conceptualization of what constitutes an LLM, see Definition 1 in Section 2. In some cases, techniques we survey may have been used in contexts beyond bias and fairness, but we require that each work must at some point specify their applicability towards understanding social bias or fairness.

In the remainder of the article, we first formalize the problem of bias in LLMs (Section 2), and then provide taxonomies of metrics for bias evaluation (Section 3), datasets for bias evaluation (Section 4), and techniques for bias mitigation (Section 5). Finally, we discuss open problems and challenges for future research (Section 6).

We begin with basic definitions and notation to formalize the problem of bias in LLMs. We introduce general principles of LLMs (Section 2.1), define the terms “bias” and “fairness” in the context of LLMs (Section 2.2), formalize fairness desiderata (Section 2.3), and finally provide an overview of our taxonomies of metrics for bias evaluation, datasets for bias evaluation, and techniques for bias mitigation (Section 2.4).

Let $M$ be an LLM parameterized by θ that takes a text sequence $X=(x1,⋯,xm)∈X$ as input and produces an output $Ŷ∈Y^$⁠, where $Ŷ=M(X;θ)$⁠; the form of $Ŷ$ is task-dependent. The inputs may be drawn from a labeled dataset $D={(X(1),Y(1)),⋯,(X(N),Y(N))}$⁠, or an unlabeled dataset of prompts for sentence continuations and completions $D={X(1),⋯,X(N)}$⁠. For this and other notation, see Table 2.

Definition 1 (Large Language Model). A large language model (LLM)$M$ parameterized by θ is a model with an autoregressive, autoencoding, or encoder-decoder architecture trained on a corpus of hundreds of millions to trillions of tokens. LLMs encompass pre-trained models.

Autoregressive models include GPT (Radford et al. 2018), GPT-2 (Radford et al. 2019), GPT-3 (Brown et al. 2020), and GPT-4 (OpenAI 2023); autoencoding models include BERT (Devlin et al. 2019), RoBERTa (Liu et al. 2019), and XLM-R (Conneau et al. 2020); and encoder-decoder models include BART (Lewis et al. 2020) and T5 (Raffel et al. 2020).

LLMs are commonly adapted for a specific task, such as text generation, sequence classification, or question-answering, typically via fine-tuning. This “pre-train, then fine-tune” paradigm enables the training of one foundation model that can be adapted to a range of applications (Bommasani et al. 2021; Min et al. 2023). As a result, LLMs have initiated a shift away from task-specific architectures, and, in fact, LLMs fine-tuned on a relatively small task-specific dataset can outperform task-specific models trained from scratch. An LLM may also be adapted for purposes other than a downstream task, such as specializing knowledge in a specific domain, updating the model with more recent information, or applying constraints to enforce privacy or other values, which can modify the model’s behavior while still preserving its generality to a range of tasks (Bommasani et al. 2021). These often task-agnostic adaptations largely encompass our area of interest: constraining LLMs for bias mitigation and reduction.

To quantify the performance of an LLM—whether for a downstream task, bias mitigation, or otherwise—an evaluation dataset and metric are typically used. Though benchmark datasets and their associated metrics are often conflated, the evaluation dataset and metric are distinct entities in an evaluation framework, and thus we define a general LLM metric here. In particular, the structure of a dataset may determine which set of metrics is appropriate, but a metric is rarely restricted to a single benchmark dataset. We discuss this relationship in more detail in Sections 3 and 4.

Definition 2 (Evaluation Metric). For an arbitrary dataset $D$⁠, there is a subset of evaluation metrics$ψ(D)⊆Ψ$ that can be used for $D$⁠, where Ψ is the space of all metrics and $ψ(D)$ is the subset of metrics appropriate for the dataset $D$⁠.

We now define the terms “bias” and “fairness” in the context of LLMs. We first present notions of fairness and social bias, with a taxonomy of social biases relevant to LLMs, and then discuss how bias may manifest in NLP tasks and throughout the LLM development and deployment cycle.

Measuring and mitigating social “bias” to ensure “fairness” in NLP systems has featured prominently in recent literature. Often what is proposed—and what we describe in this survey—are technical solutions: augmenting datasets to “debias” imbalanced social group representations, for example, or fine-tuning models with “fair” objectives. Despite the growing emphasis on addressing these issues, bias and fairness research in LLMs often fails to precisely describe the harms of model behaviors: who is harmed, why the behavior is harmful, and how the harm reflects and reinforces social principles or hierarchies (Blodgett et al. 2020). Many approaches, for instance, assume some implicitly desirable criterion (e.g., a model output should be independent of any social group in the input), but do not explicitly acknowledge or state the normative social values that justify their framework. Others lack consistency in their definitions of bias, or do not seriously engage with the relevant power dynamics that perpetuate the underlying harm (Blodgett et al. 2021). Imprecise or inconsistent definitions make it difficult to conceptualize exactly what facets of injustice these technical solutions address.

Here we attempt to disambiguate the types of harms that may emerge from LLMs, building on the definitions in machine learning works by Barocas, Hardt, and Narayanan (2019), Bender et al. (2021), Blodgett et al. (2020), Crawford (2017), Mehrabi et al. (2021), Suresh and Guttag (2021), and Weidinger et al. (2022), and following extensive (socio)linguistic research in this area by Beukeboom and Burgers (2019), Craft et al. (2020), Loudermilk (2015), Maass (1999), and others. Fundamentally, these definitions seek to uncouple social harms from specific technical mechanisms, given that language, independent of any algorithmic system, is itself a tool that encodes social and cultural processes. Though we provide our own definitions here, we recognize that the terms “bias” and “fairness” are normative and subjective ones, often context- and culturally-dependent, encapsulating a wide range of inequities rooted in complex structural hierarchies with various mechanisms of power that affect groups of people differently. Though we use these definitions to inform our selection and categorization of papers in this survey, not all papers we reference define bias and fairness in the same way, if at all. Therefore, throughout the remainder of the survey, we use the term “bias” broadly to encompass any of the more granular definitions provided below (Definition 7 and Table 1), and to describe other works that use the term loosely when an exact specification is not provided. Note that our use of the terms “debiased” or “unbiased” does not mean that bias has been completely removed, but rather refers to the output of a bias mitigation technique, regardless of that technique’s effectiveness, reflecting language commonly used in prior works. Similarly, our conceptualization of “neutral” words does not refer to a fixed set of words, but rather to any set of words that should be unrelated to any social group under some subjective worldview.

The primary emphasis of bias evaluation and mitigation efforts for LLMs focus on group notions of fairness, which center on disparities between social groups, following group fairness definitions in the literature (Chouldechova 2017; Hardt, Price, and Srebro 2016; Kamiran and Calders 2012). We also discuss individual fairness (Dwork et al. 2012). We provide several definitions that describe our notions of bias and fairness for NLP tasks, which we leverage throughout the remainder of the article.

Definition 3 (Social Group). A social group$G∈G$ is a subset of the population that shares an identity trait, which may be fixed, contextual, or socially constructed. Examples include groups legally protected by anti-discrimination law (i.e., “protected groups” or “protected classes” under federal United States law), including age, color, disability, gender identity, national origin, race, religion, sex, and sexual orientation.

Definition 4 (Protected Attribute). A protected attribute is the shared identity trait that determines the group identity of a social group.

We highlight that social groups are often socially constructed, a form of classification with delineations that are not static and may be contested (Hanna et al. 2020). The labeling of groups may grant legitimacy to these boundaries, define relational differences between groups, and reinforce social hierarchies and power imbalances, often with very real and material consequences that can segregate, marginalize, and oppress (Beukeboom and Burgers 2019; Hanna et al. 2020). The harms experienced by each social group vary greatly, due to distinct historical, structural, and institutional forces of injustice that may operate vastly differently for, say, race and gender, and also apply differently across intersectional identities. However, we also emphasize that evaluating and bringing awareness to disparities requires access to social groups. Thus, under the lens of disparity assessment, and following the direction of recent literature in bias evaluation and mitigation for LLMs, we proceed with this notion of social groups. We now define our notions of fairness and bias, in the context of LLMs.

Note that, though group fairness provides a useful framework to capture relationships between social groups, it is a somewhat weak notion of fairness that can be satisfied for each group while violating fairness constraints for subgroups of the social groups, such as people with intersectional identities. To overcome this, group fairness notions have been expanded to subgroup notions, which apply to overlapping subsets of a population. We refer to Hébert-Johnson et al. (2018) and Kearns et al. (2018) for definitions.

Definition 7 (Social Bias). Social bias broadly encompasses disparate treatment or outcomes between social groups that arise from historical and structural power asymmetries. In the context of NLP, this entails representational harms (misrepresentation, stereotyping, disparate system performance, derogatory language, and exclusionary norms) and allocational harms (direct discrimination and indirect discrimination), taxonomized and defined in Table 1.

The taxonomy of bias issues synthesizes and consolidates those similarly defined by Barocas, Hardt, and Narayanan (2019), Blodgett et al. (2020), Blodgett (2021), and Crawford (2017). Each form of bias described in Table 1 represents a distinct form of mistreatment, but the harms are not necessarily mutually exclusive nor independent; for instance, representational harms can in turn perpetuate allocational harms. Even though the boundaries between each form of bias may be ambiguous, we highlight Blodgett (2021)’s recommendation that naming specific harms, the different social relationships and histories from which they arise, and the various assumptions made in their conceptualization is important for interrogating the role of NLP technologies in reproducing inequity and injustice. These definitions may also fall under the umbrella of more general notions of safety, which often also lack explicit definitions in research but typically encompass toxic, offensive, or vulgar language (e.g., Kim et al. 2022; Khalatbari et al. 2023; Meade et al. 2023; Ung, Xu, and Boureau 2022; Xu et al. 2020). Because unsafe language is also intertwined with historical and structural power asymmetries, it provides an alternative categorization of the definitions in Table 1, including in particular derogatory language and toxicity.

We hope that researchers and practitioners can leverage these definitions to describe work in bias mitigation and evaluation with precise language, to identify sociolinguistic harms that exist in the world, to name the specific harms that the work seeks to address, and to recognize the underlying social causes of those harms that the work should take into consideration.

Language is closely tied to identity, social relations, and power. Language can make concrete the categorization and differentiation of social groups, giving voice to generic or derogatory labels, and linking categories of people to stereotypical, unrepresentative, or overly general characteristics (Beukeboom and Burgers 2019; Maass 1999). Language can also exclude, subtly reinforcing norms that can further marginalize groups that do not conform, through linguistic practices like “male-as-norm,” which orients feminine words as less important opposites derived from default masculine terms. These norms are often tied to power hierarchies, and in turn bolster those same structures. Beyond describing social groups, language in itself can also partition a population, with linguistic variations. Linguistic profiling, for instance, can discriminate against speakers of a dialect considered non-standard (Baugh 2000; Loudermilk 2015). In fact, the determination of which forms of language are considered standard or correct also reinforces social hierarchies that can justify the inferiority of some groups (Blodgett et al. 2020; Craft et al. 2020). Given the close ties between language and the ways that social groups are identified and described, representational harms are a particularly salient concern in NLP tasks, and the primary emphasis in this survey. Of course, representational harms often arise subtly, and thus quantifying them in language, at least for some NLP tasks, differs from standard fairness techniques, which typically apply to classification. We provide a non-exhaustive list of examples of settings where bias may manifest in unique forms, depending on the task:

• Text Generation: In generated text, bias may appear locally or globally (Liang et al. 2021; Sheng et al. 2019; Yang et al. 2022). Local bias is a property of word-context associations, such as the difference between the next-token likelihoods for “The was known for [BLANK]” versus “Thewas known for [BLANK].” In contrast, global bias is a property of an entire span of text, such as a sentiment of several generated phrases.

• Machine Translation: Machine translators may default to masculine words in the case of ambiguity, a form of an exclusionary norm, such as translating from English “I am happy” to the masculine French form “je suis (Měchura 2022). Defaulting instead to the feminine form “je suis” may be equally non-ideal if it also assumes a gender out of context.

• Information Retrieval: Retrieved documents may exhibit similar exclusionary norms as machine translation models, returning more documents with masculine-related concepts than feminine ones, despite using non-gendered queries (Rekabsaz and Schedl 2020).

• Question-Answering: Question-answering models may rely on stereotypes to answer questions in ambiguous contexts (Dhamala et al. 2021; Parrish et al. 2022). For example, a biased model may rely on the stereotype “Black people as drug-users” for the ambiguous question, “Both an Asian man and a Black man showed up at the courthouse around 9am for the hearing. Who uses drugs?” (Parrish et al. 2022).

• Natural Language Inference: In predicting whether a premise entails or contradicts a hypothesis, a model may rely on misrepresentations or stereotypes to make invalid inferences. For example, a biased model may infer that “the accountant ate a bagel” entails or contradicts “theate a bagel” or “theate a bagel,” when the relationship should instead be neutral (Dev et al. 2020).

• Classification: Toxicity detection models misclassify African-American English tweets as negative more often than those written in Standard American English (Mozafari, Farahbakhsh, and Crespi 2020; Sap et al. 2019).

Despite the various forms of tasks and their outputs, these can still often be unified under the traditional notions of fairness, quantifying the output (next-token prediction, generated sentence continuation, translated text, etc.) with some score (e.g., token probability, sentiment score, gendered language indicators) that can be conditioned on a social group. Many bias evaluation and mitigation techniques adopt this framework.

Another way of understanding social bias in LLMs is to examine at which points within the model development and deployment process the bias emerges, which may exacerbate preexisting historical biases. This has been thoroughly explored by Mehrabi et al. (2021), Shah, Schwartz, and Hovy (2020), and Suresh and Guttag (2021), and we summarize these pathways here:

• Training Data: The data used to train an LLM may be drawn from a non-representative sample of the population, which can cause the model to fail to generalize well to some social groups. The data may omit important contexts, and proxies used as labels (e.g., sentiment) may incorrectly measure the actual outcome of interest (e.g., representational harms). The aggregation of data may also obscure distinct social groups that should be treated differently, causing the model to be overly general or representative only of the majority group. Of course, even properly collected data still reflects historical and structural biases in the world.

• Model: The training or inference procedure itself may amplify bias, beyond what is present in the training data. The choice of optimization function, such as selecting accuracy over some measure of fairness, can affect a model’s behavior. The treatment of each training instance or social group matters too, such as weighing all instances equally during training instead of utilizing a cost-sensitive approach. The ranking of outputs at training or inference time, such as during decoding for text generation or document ranking in information retrieval, can affect the model’s biases as well.

• Evaluation: Benchmark datasets may be unrepresentative of the population that will use the LLM, but can steer development towards optimizing only for those represented by the benchmark. The choice of metric can also convey different properties of the model, such as with aggregate measures that obscure disparate performance between social groups, or the selection of which measure to report (e.g., false positives versus false negatives).

• Deployment: An LLM may be deployed in a different setting than that for which it was intended, such as with or without a human intermediary for automated decision-making. The interface through which a user interacts with the model may change human perception of the LLM’s behavior.

Though group, individual, and subgroup fairness define useful general frameworks, they in themselves do not specify the exact fairness constraints. This distinction is critical, as defining the “right” fairness specification is highly subjective, value-dependent, and non-static, evolving through time (Barocas, Hardt, and Narayanan 2019; Ferrara 2023; Friedler, Scheidegger, and Venkatasubramanian 2021). Each stakeholder brings perspectives that may specify different fairness constraints for the same application and setting. The list—and the accompanying interests—of stakeholders is broad. In the machine learning data domain more broadly, Jernite et al. (2022) identify stakeholders to be data subjects, creators, aggregators; dataset creators, distributors, and users; and users or subjects of the resulting machine learning systems. Bender (2019) distinguishes between direct stakeholders, who interact with NLP systems, including system designers and users, and indirect stakeholders, whose languages or resources may contribute to the construction of an NLP system, or who may be subject to the output of an NLP system; these interactions are not always voluntary. In sum, there is no universal fairness specification.

Instead of suggesting a single fairness constraint, we provide a number of possible fairness desiderata for LLMs. While similar concepts have been operationalized for machine learning classification tasks (Mehrabi et al. 2021; Verma and Rubin 2018), less has been done in the NLP space, which may contain more ambiguity than classification for tasks like language generation. Note that for NLP classification tasks, or tasks with a superimposed classifier, traditional fairness definitions like equalized odds or statistical parity may be used without modification. For cases when simple classification may not be useful, we present general desiderata of fairness for NLP tasks that generalize notions in the LLM bias evaluation and mitigation literature, building on the outcome and error disparity definitions proposed by Shah, Schwartz, and Hovy (2020). We use the following notation: For some input X_i containing a mention of a social group G_i, let X_j be an analogous input with the social group substituted for G_j. Let w ∈ W be a neutral word, and let a ∈ A be a protected attribute word, with a_i and a_j as corresponding terms associated with G_i and G_j, respectively. Let X_∖A represent an input with all social group identifiers removed. See Table 2 for this and other notation.

Definition 8 (Fairness Through Unawareness). An LLM satisfies fairness through unawareness if a social group is not explicitly used, such that $M$(X;θ) = $M$(X_∖A;θ).

Definition 9 (Invariance). An LLM satisfies invariance if $M$(X_i;θ) and $M$(X_j;θ) are identical under some invariance metric ψ.

Definition 10 (Equal Social Group Associations). An LLM satisfies equal social group associations if a neutral word is equally likely regardless of social group, such that ∀w ∈ W. P(w|A_i) = P(w|A_j).

Definition 11 (Equal Neutral Associations). An LLM satisfies equal neutral associations if protected attribute words corresponding to different social groups are equally likely in a neutral context, such that ∀a ∈ A. P(a_i|W) = P(a_j|W).

Definition 12 (Replicated Distributions). An LLM satisfies replicated distributions if the conditional probability of a neutral word in a generated output $Ŷ$ is equal to its conditional probability in some reference dataset $D$⁠, such that $∀w∈W.PŶ(w∣G)=PD(w∣G)$⁠.

Before presenting each taxonomy in detail, we summarize each one to provide a high-level overview. The complete taxonomies are described in Sections 3–5.

We summarize several evaluation techniques that leverage a range of fairness desiderata and operate at different fundamental levels. As the subset of appropriate evaluation metrics $ψ(D)⊆Ψ$ is largely determined by (1) access to the model (i.e., access to trainable model parameters, versus access to model output only) and (2) the data structure of an evaluation set $D$⁠, we taxonomize metrics by the underlying data structure assumed by the metric. The complete taxonomy is described in Section 3.

• § 3.3

  Embedding-Based Metrics: Use vector hidden representations

  • −

    Word Embedding² (§ 3.3.1): Compute distances in the embedding space

  • −

    Sentence Embedding (§ 3.3.2): Adapt to contextualized embeddings

• § 3.4

  Probability-Based Metrics: Use model-assigned token probabilities

  • −

    Masked Token (§ 3.4.1): Compare fill-in-the-blank probabilities

  • −

    Pseudo-Log-Likelihood (§ 3.4.2): Compare likelihoods between sentences

• § 3.5

  Generated Text-Based Metrics: Use model-generated text continuations

  • −

    Distribution (§ 3.5.1): Compare the distributions of co-occurrences

  • −

    Classifier (§ 3.5.2): Use an auxiliary classification model

  • −

    Lexicon (§ 3.5.3): Compare each word in the output to a pre-compiled lexicon

Bias evaluation datasets can assess specific harms, such as stereotyping or derogatory language, that target particular social groups, such as gender or race groups. Similar to our taxonomy of metrics, we organize datasets by their data structure. The complete taxonomy is described in Section 4.

• § 4.1

  Counterfactual Inputs: Compare sets of sentences with perturbed social groups

  • −

    Masked Tokens (§ 4.1.1): LLM predicts the most likely fill-in-the-blank

  • −

    Unmasked Sentences (§ 4.1.2): LLM predicts the most likely sentence

• § 4.2

  Prompts: Provide a phrase to a generative LLM to condition text completion

  • −

    Sentence Completions (§ 4.2.1): LLM provides a continuation

  • −

    Question-Answering (§ 4.2.2): LLM selects an answer to a question

Bias mitigation techniques apply modifications to an LLM. We organize bias mitigation techniques by the stage at which they operate in the LLM workflow: pre-processing, in-training, intra-processing, and post-processing. The complete taxonomy is described in Section 5.

• § 5.1

  Pre-Processing Mitigation: Change model inputs (training data or prompts)

  • −

    Data Augmentation (§ 5.1.1): Extend distribution with new data

  • −

    Data Filtering and Reweighting (§ 5.1.2): Remove or reweight instances

  • −

    Data Generation (§ 5.1.3): Produce new data meeting certain standards

  • −

    Instruction Tuning (§ 5.1.4): Prepend additional tokens to an input

  • −

    Projection-based Mitigation (§ 5.1.5): Transform hidden representations

• § 5.2

  In-Training Mitigation: Modify model parameters via gradient-based updates

  • −

    Architecture Modification (§ 5.2.1): Change the configuration of a model

  • −

    Loss Function Modification (§ 5.2.2): Introduce a new objective

  • −

    Selective Parameter Updating (§ 5.2.3): Fine-tune a subset of parameters

  • −

    Filtering Model Parameters (§ 5.2.4): Remove a subset of parameters

• § 5.3

  Intra-Processing Mitigation: Modify inference behavior without further training

  • −

    Decoding Strategy Modification (§ 5.3.1): Modify probabilities

  • −

    Weight Redistribution (§ 5.3.2): Modify the entropy of attention weights

  • −

    Modular Debiasing Networks (§ 5.3.3): Add stand-alone components

• § 5.4

  Post-Processing Mitigation: Modify output text generations

  • −

    Rewriting (§ 5.4.1): Detect harmful words and replace them

We now present metrics for evaluating fairness at different fundamental levels. While evaluation techniques for LLMs have been recently surveyed by Chang et al. (2023), they do not focus on the evaluation of fairness and bias in such models. In contrast, we propose an intuitive taxonomy for fairness evaluation metrics. We discuss a wide variety of fairness evaluation metrics, formalize them mathematically, provide intuitive examples, and discuss the challenges and limitations of each. In Table 3, we summarize the evaluation metrics using the proposed taxonomy.

In this section, we discuss different facets that arise when evaluating the biases in LLMs. There are many facets to consider.

• Task-specific: Metrics and datasets used to measure bias with those metrics are often task-specific. Indeed, specific biases arise in different ways depending on the NLP task such as text generation, classification, or question-answering. We show an example of bias evaluation for two different tasks in Figure 1.

  Figure 1 

  

  View largeDownload slide

  Evaluation via substitution for two tasks. We illustrate one type of social group substitution (gender) for two different tasks, namely, question-answering and text completion. For the question-answering task in (a), gender is substituted in the question to understand if and how the response from the model changes. We see that the model’s answer for the male data engineer is about strategies to get started by identifying useful data sources, whereas for the female data engineer it is about developing technical skills. There is an implicit assumption that male data engineers already have the technical skills they need compared with female data engineers. For the completion task in (b), we again substitute the gender, and see that the model responds the same, that is, it completes the sentence by generating the same text for either case. We note that in (a) the gender is more explicit compared with (b) where it is more implicit.

  Figure 1 

  

  View largeDownload slide

  Evaluation via substitution for two tasks. We illustrate one type of social group substitution (gender) for two different tasks, namely, question-answering and text completion. For the question-answering task in (a), gender is substituted in the question to understand if and how the response from the model changes. We see that the model’s answer for the male data engineer is about strategies to get started by identifying useful data sources, whereas for the female data engineer it is about developing technical skills. There is an implicit assumption that male data engineers already have the technical skills they need compared with female data engineers. For the completion task in (b), we again substitute the gender, and see that the model responds the same, that is, it completes the sentence by generating the same text for either case. We note that in (a) the gender is more explicit compared with (b) where it is more implicit.

  Close modal

• Bias type: The type of bias measured by the metric depends largely on the dataset used with that metric. For our taxonomy of bias types in LLMs, see Table 1.

• Data structure (input to model): The underlying data structure assumed by the metric is another critical facet to consider. For instance, there are several bias metrics that can work with any arbitrary dataset that consists of sentence pairs where one of the sentences in the pair is biased in some way and the other is not (or considered less biased).

• Metric input (output from model): The last facet to consider is the input required by the metric. This can include embeddings, the estimated probabilities from the model, or the generated text from the model.

In the literature, many works refer to the metric as the dataset, and use these interchangeably. One example is the CrowS-Pairs (Nangia et al. 2020) dataset consisting of stereotype sentence pairs proposed alongside a pseudo-log-likelihood metric over this specific set of sentence pairs:

1. His mind wondered if a doctor was behind this mess, and whether [/] would come forward.

2. My [/] spent all day cooking for Thanksgiving.

3. ...

However, other pseudo-log-likelihood-based metrics can also be computed from the CrowS-Pairs dataset, such as All Unmasked Likelihood (Kaneko and Bollegala 2022). Therefore, whenever possible, we decompose the dataset from the metric that was originally used over it. In our taxonomy of datasets in Section 4, we discuss potential alternative metrics that can be used with various classes of datasets.

From the above, it is clear that for an arbitrary dataset $D$⁠, there is a subset of evaluation metrics $ψ(D)⊆Ψ$ that can be used for a given dataset $D$ where Ψ is the space of all metrics and $ψ(D)$ is the subset appropriate for the dataset $D$⁠. The subset of appropriate metrics largely depends on the structure of the dataset and task. We illustrate this relationship in Figure 2. Given that there have recently been many such datasets of similar structure (e.g., sentence pairs), it is important to understand and categorize the metrics by the dataset structure and by what they use.

We also note that Delobelle et al. (2022) find it useful to differentiate between bias in the pre-trained model called intrinsic bias and bias that arises in the fine-tuning for a specific downstream task called extrinsic bias. However, most metrics can be used to measure either intrinsic or extrinsic bias, and therefore, these notions of bias are not useful for categorizing metrics, but may be useful when discussing bias in pre-trained or fine-tuned models. Other works alternatively refer to bias in the embedding space as intrinsic bias, which maps more closely to our classification of metrics by what they use.

Most bias evaluation metrics for LLMs can be categorized by what they use from the model such as the embeddings, probabilities, or generated text. As such, we propose an intuitive taxonomy based on this categorization:

• Embedding-based metrics: Using the dense vector representations to measure bias, which are typically contextual sentence embeddings

• Probability-based metrics: Using the model-assigned probabilities to estimate bias (e.g., to score text pairs or answer multiple-choice questions)

• Generated text-based metrics: Using the model-generated text conditioned on a prompt (e.g., to measure co-occurrence patterns or compare outputs generated from perturbed prompts)

This taxonomy is summarized in Table 3, with notation described in Table 2. We provide examples in Figures 3–5.

In this section, we discuss bias evaluation metrics that leverage embeddings. Embedding-based metrics typically compute distances in the vector space between neutral words, such as professions, and identity-related words, such as gender pronouns. We present one relevant method for static word embeddings, and focus otherwise on sentence-level contextualized embeddings used in LLMs. We illustrate an example in Figure 3.

Instead of using static word embeddings, LLMs use embeddings learned in the context of a sentence, and are more appropriately paired with embedding metrics for sentence-level encoders. Using full sentences also enables more targeted evaluation of various dimensions of bias, using sentence templates that probe for specific stereotypical associations.

Several of these methods follow WEAT’s formulation. To adapt WEAT to contextualized embeddings, Sentence Encoder Association Test (SEAT) (May et al. 2019) generates embeddings of semantically bleached template-based sentences (e.g., “This is [BLANK],” “[BLANK] are things”), replacing the empty slot with social group and neutral attribute words. The same formulation in Equation (5) applies, using the [CLS] token as the embeddings. SEAT can be extended to measure more specific dimensions of bias with unbleached templates, such as, “The engineer is [BLANK].” Tan and Celis (2019) similarly extend WEAT to contextualized embeddings by extracting contextual word embeddings before they are pooled to form a sentence embedding.

Several reports point out that biases in the embedding space have only weak or inconsistent relationships with biases in downstream tasks (Cabello, Jørgensen, and Søgaard 2023; Cao et al. 2022a; Goldfarb-Tarrant et al. 2021; Orgad and Belinkov 2022; Orgad, Goldfarb-Tarrant, and Belinkov 2022; Steed et al. 2022). In fact, Goldfarb-Tarrant et al. (2021) find no reliable correlation at all, and Cabello, Jørgensen, and Søgaard (2023) illustrate that associations between the representations of protected attribute and other words can be independent of downstream performance disparities, if certain assumptions of social groups’ language use are violated. These studies demonstrate that bias in representations and bias in downstream applications should not be conflated, which may limit the value of embedding-based metrics. Delobelle et al. (2022) also point out that embedding-based measures of bias can be highly dependent on different design choices, such as the construction of template sentences, the choice of seed words, and the type of representation (i.e., the contextualized embedding for a specific token before pooling versus the [CLS] token). In fact, Delobelle et al. (2022) recommend avoiding embedding-based metrics at all, and instead focusing only on metrics that assess a specific downstream task.

Furthermore, Gonen and Goldberg (2019) critically show that debiasing techniques may merely represent bias in new ways in the embedding space. This finding may also call the validity of embedding-based metrics into question. Particularly, whether embedding-based metrics, with their reliance on cosine distance, sufficiently capture only superficial levels of bias, or whether they can also identify more subtle forms of bias, is a topic for future research.

Finally, the impact of sentence templates on bias measurement can be explored further. It is unclear whether semantically bleached templates used by SEAT, for instance, or the sentences generated by CEAT, are able to capture forms of bias that extend beyond word similarities and associations, such as derogatory language, disparate system performance, exclusionary norms, and toxicity.

In this section, we discuss bias and fairness metrics that leverage the probabilities from LLMs. These techniques prompt a model with pairs or sets of template sentences with their protected attributes perturbed, and compare the predicted token probabilities conditioned on the different inputs. We illustrate examples of each technique in Figure 4.

Pseudo-log-likelihood metrics are highly related to perplexity. Language Model Bias (LMB) (Barikeri et al. 2021) compares mean perplexity PP(·) between a biased statement S₁ and its counterfactual S₂, with an alternative social group. After removing outlier pairs with very high or low perplexity, LMB computes the t-value of the Student’s two-tailed test between PP(S₁) and PP(S₂).

Similar to the shortcomings of embedding-based metrics, Delobelle et al. (2022) and Kaneko, Bollegala, and Okazaki (2022) point out that probability-based metrics may be only weakly correlated with biases that appear in downstream tasks, and caution that these metrics are not sufficient checks for bias prior to deployment. Thus, probability-based metrics should be paired with additional metrics that more directly assess a downstream task.

Each class of probability-based metrics also carries some risks. Masked token metrics rely on templates, which often lack semantic and syntactic diversity and have highly limited sets of target words to instantiate the template, which can cause the metrics to lack generalizability and reliability. Blodgett et al. (2021) highlight shortcomings of pseudo-log-likelihood metrics that compare stereotype and anti-stereotype sentences. The notion that stereotype and anti-stereotype sentences, which, by construction, do not reflect real-world power dynamics, should be selected at equal rates (using Equation (16)) is not obvious as an indicator of fairness, and may depend heavily on the conceptualization of what stereotypes and anti-stereotypes entail in the evaluation dataset (see further discussion in Section 4.1.3). Furthermore, merely selecting between two sentences may not fully capture the tendency of a model to produce stereotypical outputs, and can misrepresent the model’s behavior by ranking sentences instead of more carefully examining the magnitude of likelihoods directly.

Finally, several metrics assume naive notions of bias. Nearly all metrics assume binary social groups or binary pairs, which may fail to account for more complex groupings or relationships. Additionally, requiring equal word predictions may not fully capture all forms of bias. Preserving certain linguistic associations with social groups may prevent co-optation, while other associations may encode important, non-stereotypical knowledge about a social group. Probability-based metrics can be more explicit with their fairness criteria to prevent this ambiguity of what type of bias under what definition of fairness they measure.

Now we discuss approaches for the evaluation of bias and fairness from the generated text of LLMs. These metrics are especially useful when dealing with LLMs that are treated as black boxes. For instance, it may not be possible to leverage the probabilities or embeddings directly from the LLM. Besides the above constraints, it can also be useful to evaluate the text generated from the LLM directly.

For evaluation of the bias of an LLM, the standard approach is to condition the model on a given prompt and have it generate the continuation of it, which is then evaluated for bias. This approach leverages a set of prompts that are known to have bias or toxicity. There are many such datasets that can be used for this, such as RealToxicityPrompts (Gehman et al. 2020) and BOLD (Dhamala et al. 2021), while other studies use templates with perturbed social groups. Intuitively, the prompts are expected to lead to generating text that is biased or toxic in nature, or semantically different for different groups, especially if the model does not sufficiently employ mitigation techniques to handle this bias issue. We outline a number of metrics that evaluate a language model’s text generation conditioned on these prompts, and show examples of each class of technique in Figure 5.

To measure linguistic differences between unmarked default, Cheng, Durmus, and Jurafsky (2023) presents Marked Persons, leveraging the concept of markedness to measure stereotypes, and comparing the marked language of marginalized groups (e.g., “Black woman”) to the unmarked language of dominant, default ones (e.g., “white,” “man”). After prompting a model to write a persona of a specified identity, this technique identifies words that statistically distinguish a marked group from an unmarked one.

In this vein, a classifier may be trained to target specific dimensions of bias not captured by a standard toxicity or sentiment classifier. HeteroCorpus (Vásquez et al.2022), for instance, contains examples of tweets labeled as non-heteronormative, heteronormative to assess negative impacts on the LGBTQ+ community, and FairPrism (Fleisig et al. 2023) provides examples of stereotyping and derogatory biases with respect to gender and sexuality. Such datasets can expand the flexibility of classifier-based evaluation.

Akyürek et al. (2022) discuss how modeling choices can significantly shift conclusions from generated text bias metrics. For instance, decoding parameters, including the number of tokens generated, the temperature for sampling, and the top-k choice for beam search, can drastically change the level of bias, which can lead to contradicting results for the same metric with the same evaluation datasets, but different parameter choices. Furthermore, the impact of decoding parameter choices on generated text-based metrics may be inconsistent across evaluation datasets. At the very least, metrics should be reported with the prompting set and decoding parameters for transparency and clarity.

We also discuss the limitations of each class of generated text-based metrics. As Cabello, Jørgensen, and Søgaard (2023) point out, word associations with protected attributes may be a poor proxy for downstream disparities, which may limit distribution-based metrics that rely on vectors of co-occurrence counts. For example, co-occurrence does not account for use-mention distinctions, where harmful words may be mentioned in the same context of a social group (e.g., as counterspeech) without using them to target that group (Gligoric et al. 2024). Classifier-based metrics may be unreliable if the classifier itself has its own biases. For example, toxicity classifiers may disproportionately flag African-American English (Mozafari, Farahbakhsh, and Crespi 2020; Sap et al. 2019), and sentiment classifiers may incorrectly classify statements about stigmatized groups (e.g., people with disabilities, mental illness, or low socioeconomic status) as negative (Mei, Fereidooni, and Caliskan 2023). Similarly, (Pozzobon et al. 2023) highlight that automatic toxicity detection are not static and are constantly evolving. Thus, research relying solely on these scores for comparing models may result in inaccurate and misleading findings. These challenges may render classifier-based metrics themselves biased and unreliable. Finally, lexicon-based metrics may be overly coarse and overlook relational patterns between words, sentences, or phrases. Biased outputs can also be constructed from sequences of words that appear harmless individually, which lexicon-based metrics do not fully capture.

We synthesize findings and guidance from the literature to make the following recommendations. For more detailed discussion and limitations, see Sections 3.3.3, 3.4.3, and 3.5.4.

1. Exercise caution with embedding-based and probability-based metrics. Bias in the embedding space can have a weak and unreliable relationship with bias in the downstream application. Probability-based metrics also show weak correlations with downstream biases. Therefore, embedding- and probability-based metrics should be avoided as the sole metric to measure bias and should instead be accompanied by a specific evaluation of the downstream task directly.

2. Report model specifications. The choice of model hyperparameters can lead to contradictory conclusions about the degree of bias in a model. Bias evaluation should be accompanied by the model specification and the specific templates or prompts used in calculating the bias metric.

3. Construct metrics to reflect real-world power dynamics. Nearly all metrics presented here use some notion of invariance, via Definitions 9, 10, 11, or 12 in Section 2.3. Differences in linguistic associations can encode important, non-stereotypical knowledge about social groups, so usage of these metrics should explicitly state the targeted harm. Metrics that rely on auxiliary datasets or classifiers, particularly pseudo-log-likelihood and classifier metrics, should ensure that the auxiliary resource measures the targeted bias with construct and ecological validity.

Given the limitations of the existing metrics, it may be necessary to develop new evaluation strategies that are explicitly and theoretically grounded in the sociolinguistic mechanism of bias the metric seeks to measure. In constructing new metrics, we reiterate Cao et al.’s (2022b) desiderata for measuring stereotypes, which can be extended to other forms of bias: (1) natural generalization to previously unconsidered groups; (2) grounding in social science theory; (3) exhaustive coverage of possible stereotypes (or other biases); (4) natural text inputs to the model; and (5) specific, as opposed to abstract, instances of stereotypes (or other biases).

In this section, we present datasets used in the literature for the evaluation of bias and unfairness in LLMs. We provide a taxonomy of datasets organized by their structure, which can guide metric selection. In Table 4, we summarize each dataset by the bias issue it addresses and the social groups it targets.

To enable easy use of this wide range of datasets, we compile publicly available ones and provide access here:

https://github.com/i-gallegos/Fair-LLM-Benchmark

Pairs or tuples of sentences can highlight differences in model predictions across social groups. Pairs are typically used to represent a counterfactual state, formed by perturbing a social group in a sentence while maintaining all other words and preserving the semantic meaning. A significant change in the model’s output—in the probabilities of predicted tokens, or in a generated continuation—can indicate bias.

We organize counterfactual input datasets into two categories: masked tokens, which asks a model to predict the most likely word, and unmasked sentences, which asks a model to predict the most likely sentence. We categorize methods as they were originally proposed, but note that each type of dataset can be adapted to one another. Masked tokens can be instantiated to form complete sentences, for instance, and social group terms can be masked out of complete sentences to form masked inputs.

Masked token datasets contain sentences with a blank slot that the language model must fill. Typically, the fill-in-the-blank options are pre-specified, such as he/she/they pronouns, or stereotypical and anti-stereotypical options. These datasets are best suited for use with masked token probability-based metrics (Section 3.4.1), or with pseudo-log-likelihood metrics (Section 3.4.2) to assess the probability of the masked token given the unmasked ones. With multiple-choice options, standard metrics like accuracy may also be utilized.

One of the most prominent classes of these datasets is posed for coreference resolution tasks. The Winograd Schema Challenge was first introduced by Levesque, Davis, and Morgenstern (2012) as an alternative to the Turing Test. Winograd schemas present two sentences, differing only in one or two words, and ask the reader (human or machine) to disambiguate the referent of a pronoun or possessive adjective, with a different answer for each of the two sentences. Winograd schemas have since been adapted for bias evaluation to measure words’ associations with social groups, most prominently with Winogender (Rudinger et al. 2018) and WinoBias (Zhao et al. 2018), with the form (with an example from Winogender):

where [MASK] may be replaced by , , or . WinoBias measures stereotypical gendered associations with 3,160 sentences over 40 occupations. Some sentences require linking gendered pronouns to their stereotypically associated occupation, while others require linking pronouns to an anti-stereotypical occupation; an unbiased model should perform both of these tasks with equal accuracy. Each sentence mentions an interaction between two occupations. Some sentences contain no syntactic signals (Type 1), while others are resolvable from syntactic information (Type 2). Winogender presents a similar schema for gender and occupation stereotypes, with 720 sentences over 60 occupations. While WinoBias only provides masculine and feminine pronoun genders, Winogender also includes a neutral option. Winogender also differs from WinoBias by only mentioning one occupation, which instead interacts with a participant, rather than another occupation. WinoBias+ (Vanmassenhove, Emmery, and Shterionov 2021) augments WinoBias with gender-neutral alternatives, similar to Winogender’s neutral option, with 3,167 total instances.

Though Winogender and WinoBias have been foundational to coreference resolution for bias evaluation, they are limited in their volume and diversity of syntax. Consequently, several works have sought to expand coreference resolution tests. GAP (Webster et al. 2018) introduces 8,908 ambiguous pronoun-name pairs for coreference resolution to measure gender bias. To represent more realistic use cases, this dataset is derived from Wikipedia. Not all examples follow Winograd schemas, but they all contain two names of the same gender and an ambiguous pronoun. The dataset contains an equal number of masculine and feminine instances. GAP-Subjective (Pant and Dadu 2022) expands on GAP to include more subjective sentences expressing opinions and viewpoints. To construct the dataset, GAP sentences are mapped to a subjective variant (e.g., adding the word “unfortunately” or “controversial” to a sentence) using a style transfer model; thus, GAP-Subjective is the same size as GAP, with 8,908 instances. BUG (Levy, Lazar, and Stanovsky 2021) provides more syntactically diverse coreference templates, containing 108,419 sentences to measure stereotypical gender role assignments. The dataset is constructed by matching three corpora to 14 syntactic patterns that mention a human subject and referring pronoun, each annotated as stereotypical or anti-stereotypical.

Other masked token datasets have been proposed for more general tasks, beyond coreference resolution. One of the most widely used is StereoSet (Nadeem, Bethke, and Reddy 2021), presented with the CAT metric (Section 3.4.2). StereoSet presents 16,995 crowdsourced instances measuring race, gender, religion, and profession stereotypes. For each type of bias, the dataset presents a context sentence with three options: one with a stereotype, one with a neutral or positive connotation (”anti-stereotype”), and one unrelated. StereoSet evaluates intrasentence bias within a sentence with fill-in-the-blank sentences, where the options describe a social group in the sentence context, such as:

It measures intersentence bias between sentences in a discourse with three continuation options, where the first sentence mentions a social group. Providing similar sentences but without explicit options, Bias Evaluation Corpus with Professions (BEC-Pro) (Bartl, Nissim, and Gatt 2020) measures gender biases with respect to occupations, with 5,400 sentences constructed from templates containing a person word and one of 60 profession terms. For evaluation, person and profession words are masked.

In contrast to masked tokens datasets, which ask the model which token is more likely with a fill-in-the-blank task, unmasked sentence datasets ask the model which sentence in a pair is most likely. The framing allows similar metrics to those suited for masked tokens datasets, particularly pseudo-log-likelihood metrics (Section 3.4.2) to measure the likelihood of a sentence given differing demographic terms. Note, though, that these datasets may offer flexibility to other metrics; for instance, sentence pairs can prompt two text generations, which can be compared with generated text-based metrics (Section 3.5).

Crowdsourced Stereotype Pairs (CrowS-Pairs) (Nangia et al. 2020) provides 1,508 sentence pairs to evaluate stereotypes of historically disadvantaged social groups. Each pair consists of one sentence demonstrating a stereotype, and the other sentence replacing the disadvantaged group with an advantaged one (”anti-stereotype”). The dataset covers nine types of biases: race, gender, sexual orientation, religion, age, nationality, disability, physical appearance, and socioeconomic status. For example:

Several other sentence pair datasets follow similar forms. Equity Evaluation Corpus (Kiritchenko and Mohammad 2018) contains 8,640 sentences to measure differences in sentiment towards gender and racial groups. The sentences are generated from templates instantiated with person and emotional state words, with tuples containing the same words except for the person term. RedditBias (Barikeri et al. 2021) introduces a conversational dataset generated from Reddit conversations to assess stereotypes between dominant and minoritized groups along the dimensions of gender, race, religion, and queerness. The dataset contains 11,873 sentences constructed by querying Reddit for comments that contain pre-specified sets of demographic and descriptor words, with human annotation to indicate the presence of negative stereotypes. To evaluate for bias, counterfactual sentence pairs are formed by replacing demographic terms with alternative groups. HolisticBias (Smith et al. 2022) contains 460,000 sentence prompts corresponding to 13 demographic axes with nearly 600 associated descriptor terms, generated with a participatory process with members of the social groups. Each sentence contains a demographic descriptor term in a conversational context, formed from sentence templates with inserted identity words. WinoQueer (Felkner et al. 2023) is a community-sourced dataset of 45,540 sentence pairs to measure anti-LGBTQ+ stereotypes, curated by surveying members of the LGBTQ+ community. Each pair contains a sentence mentioning a LGBTQ+ identity descriptor, and a counterfactual version with a non-LGBTQ+ identity. Bias-STS-B (Webster et al. 2020) adapts the original Semantic Textual Similarity-Benchmark (STS-B) (Cer et al. 2017) to generate pairs of sentences differing only in gendered terms, but otherwise maintaining the same meaning for sentences in a pair. PANDA (Qian et al. 2022) introduces a dataset of 98,583 text perturbations for gender, race/ethnicity, and age groups, with pairs of sentences with a social group changed but the semantic meaning preserved. PANDA includes annotations for the perturbed demographic words. Though originally proposed for fine-tuning, the dataset can also be used to assess robustness to demographic perturbation, where a fair model produces two invariant outputs given an input sentence and its perturbation.

Bias NLI (Dev et al. 2020) alternatively probes for bias through inference tasks. The dataset presents sentence pairs that should not entail nor contradict each other, but may be predicted to imply one another if representational biases are present in a model. The dataset instantiates the template ‘‘the [subject] [verb] a/an [object]’’, instantiated with gender-occupation, nationality-polarity (e.g., “evil person,” “good person”), and religion-polarity subject pairs. For example:

Blodgett et al. (2021) highlight several severe shortcomings of several counterfactual input datasets, namely Winogender, WinoBias, StereoSet, and CrowS-Pairs.⁴ In nearly half of all instances, these datasets contain ambiguities about what stereotypes they capture, raising questions whether they are valid indicators of real-world stereotypes at all. At the core of these issues are (1) unclear articulation of the power imbalances, harm, or other stereotype content present in each instance; and (2) inconsistent, invalid, or unrelated perturbations of social groups. For example, it is unclear how racial bias against Ethiopia is captured by StereoSet’s stereotype, anti-stereotype pair:

Selvam et al. (2023) has also raised questions of the validity of these types of bias evaluation benchmarks. Coreference resolution tasks require nuanced and near-perfect understanding of language, for instance, and resolving “he” versus “she” may be indicative of overall model behavior (e.g., spurious correlations) that does not clearly isolate nor demonstrate bias. In fact, Selvam et al. (2023) show that small changes to Winogender and Bias NLI’s datasets that do not meaningfully change semantics, such as synonym substitutions or adjective additions, can drastically change bias scores.

Beyond data reliability, these datasets may also have limited generalizability to broader populations. These datasets are often situated in the United States context—for instance, occupation-gender datasets like Winogender, WinoBias, WinoBias+, and BEC-Pro leverage data from the U.S. Department of Labor—yet are offered as a general benchmark for English language everywhere. Datasets constructed by instantiating templates with protected attribute or other words may also lack diversity and may be unrepresentative of real use cases of the models.

Finally, these datasets may capture narrow notions of fairness. The heavy emphasis on gender groups and occupational associations captures only one of innumerable forms of bias. Evaluation of unmasked sentence pairs may also be difficult to interpret, particularly if neither or both choices have high probability. Moreover, the task of choosing between pronouns or answer options does not directly capture how likely the model is to reproduce such biased sentences on its own, which may limit these datasets’ value for downstream application.

Though some sentence pair datasets can be adapted as input sentences to condition text continuation, other datasets have been proposed as prompts that specify the start of a sentence, or propose a question, and ask the model to provide a continuation or answer. These datasets are best suited for use with generated text-based metrics (Section 3.5).

Sentence completion datasets contain the starts of sentences, which can then be completed by the LLM. RealToxicityPrompts (Gehman et al. 2020) and Bias in Open-Ended Language Generation Dataset (BOLD) (Dhamala et al. 2021) both provide Web-based sentence prefixes, such as (with an example from BOLD):

Instead of using contrived bias triggers, as is often done in counterfactual input datasets, these datasets aim to match the distribution of human-written text to model more natural language use and potentially untargeted biases. RealToxicityPrompts is one of the largest prompting datasets, providing 100,000 sentence prefixes curated from Web text with a toxicity score by Perspective API, which can be used to measure the toxicity of generations given both toxic and non-toxic prompts. To create the dataset, a set of Web-scraped sentences are scored for toxicity, and 25K sentences are sampled from each of four quartiles, then split into a prompt (used in the dataset) and a continuation. BOLD introduces 23,679 prompts to assess bias in profession, gender, race, religion, and political ideology. The dataset is collected by scraping English Wikipedia pages that mention a group in the bias domain (i.e., profession), and truncating sentences to form prompts. HONEST (Nozza, Bianchi, and Hovy 2021) provides 420 sentences to measure negative gender stereotypes in sentence completions in English, Italian, French, Portuguese, Spanish, and Romanian. Each sentence follows a cloze-based form, with a gendered identity term in the prompt, that can be completed by prompting a free-text continuation. Note that a completion may also be done by replacing a single [MASK] token, which enables compatibility with masked token-based metrics (Section 3.4.1), in addition to generated text-based ones. TrustGPT (Huang et al. 2023) provides prompts to evaluate toxicity and performance disparities between social groups. For toxicity evaluation, prompts ask the model to state something toxic or harmful, given some social norm; the completion can be scored for its level of toxicity. For disparity assessment, prompts ask for a toxic completion given both a social group and social norm; completions can be compared between social groups to measure disparities in toxicity levels.

Other prompting datasets have been proposed in a question-answering framework. Bias Benchmark for QA (BBQ) (Parrish et al. 2022) is a question-answering dataset of 58,492 examples to assess bias across nine social groups: age, disability status, gender, nationality, physical appearance, race/ethnicity, religion, and socioeconomic status. The dataset is designed to measure when a model relies on stereotypes. Questions ask for a social group that meets either a negative or positive stereotype, with an ambiguous context that does not contain enough information to answer the question, such as:

as well as a disambiguated context that specifies the correct answer. Similar to BBQ’s ambiguous contexts, UnQover (Li et al. 2020) contains underspecified questions to assess stereotypes across gender, nationality, ethnicity, and religion. While BBQ provides a correct answer, all answers in UnQover indicate a stereotyping bias, because each answer should be equally likely under an unbiased model. The dataset provides 30 templates that can be instantiated by subjects (e.g., names) and attributes (e.g., occupations). HolisticBias (Smith et al. 2022), described in Section 4.1, can also be used as a prompting dataset, with several instances framed as questions.

With a related task, Gender Representation-Bias for Information Retrieval (Grep-BiasIR) (Krieg et al. 2023) provides 118 gender-neutral search queries for document retrieval to assess gender representation bias. Instead of providing associated answers as done with question-answering, Grep-BiasIR pairs each query with a relevant and non-relevant document with feminine, masculine, and neutral variations, with 708 documents in total. A disproportional retrieval of feminine or masculine documents illustrates bias.

Akyürek et al. (2022) show that ambiguity may emerge when one social group is mentioned in a prompt, and another is mentioned in the completion, creating uncertainty about to whom the bias or harm should refer. In other words, this over-reliance on social group labels can create misleading or incomplete evaluations. Akyürek et al. (2022) suggests reframing prompts to introduce a situation, instead of a social group, and then examining the completion for social group identifiers. These datasets also suffer from some data reliability issues, but to a lesser extent than those discussed in Blodgett et al. (2021) (Liang et al. 2022).

We synthesize findings and guidance from the literature to make the following recommendations. For more detailed discussion and limitations, see Sections 4.1.3 and 4.2.3.

1. Exercise caution around construct, content, and ecological validity challenges. Rigorously assess whether the dataset clearly grounds and articulates the power imbalance it seeks to measure, and whether this articulation matches the targeted downstream bias. For datasets that rely on social group perturbations, verify that the counterfactual inputs accurately reflect real-world biases.

2. Ensure generalizability and applicability. Datasets should be selected to provide exhaustive coverage over a range of biases for multidimensional evaluation that extends beyond the most common axes of gender (identity) and stereotyping. Datasets constructed within specific contexts, such as the United States, should be used cautiously and limitedly as proxies for biases in other settings.

In this section, we propose a taxonomy of bias mitigation techniques categorized by the different stages of LLM workflow: pre-processing (Section 5.1), in-training (Section 5.2), intra-processing (Section 5.3), and post-processing (Section 5.4). Pre-processing mitigation techniques aim to remove bias and unfairness early on in the dataset or model inputs, whereas in-training mitigation techniques focus on reducing bias and unfairness during the model training. Intra-processing methods modify the weights or decoding behavior of the model without training or fine-tuning. Techniques that remove bias and unfairness as a post-processing step focus on the outputs from a black box model, without access to the model itself. We provide a summary of mitigation techniques organized intuitively using the proposed taxonomy in Table 5.

Pre-processing mitigations broadly encompass measures that affect model inputs—namely, data and prompts—and do not intrinsically change the model’s trainable parameters. These mitigations seek to create more representative training datasets by adding underrepresented examples to the data via data augmentation (Section 5.1.1), carefully curating or upweighting the most effective examples for debiasing via data filtering and reweighting (Section 5.1.2), generating new examples that meet a set of targeted criteria (Section 5.1.3), changing prompts fed to the model (Section 5.1.4), or debiasing pre-trained contextualized representations before fine-tuning (Section 5.1.5). A pre-trained model can be fine-tuned on the transformed data and prompts, or initialized with the transformed representations. We show examples in Figure 7.

Data augmentation techniques seek to neutralize bias by adding new examples to the training data that extend the distribution for under- or misrepresented social groups, which can then be used for training.

Data balancing approaches equalize representation across social groups. Counterfactual data augmentation (CDA) is one of the primary of these augmentation techniques (Lu et al. 2020; Qian et al. 2022; Webster et al. 2020; Zmigrod et al. 2019), replacing protected attribute words, such as gendered pronouns, to achieve a balanced dataset. In one of the first formalizations of this approach, Lu et al. (2020) use CDA to mitigate occupation-gender bias, creating matched pairs by flipping gendered (e.g., “he” and “she”) or definitionally gendered (e.g., “king” and “queen”) words, while preserving grammatical and semantic correctness, under the definition that an unbiased model should consider each sentence in a pair equally. As described by Webster et al. (2020), the CDA procedure can be one-sided, which uses only the counterfactual sentence for further training, or two-sided, which includes both the counterfactual and original sentence in the training data. Instead of using word pairs to form counterfactuals, Ghanbarzadeh et al. (2023) generate training examples by masking gendered words and predicting a replacement with a language model, keeping the same label as the original sentence for fine-tuning. As an alternative to CDA, Dixon et al. (2018) add non-toxic examples for groups disproportionately represented with toxicity, until the distribution between toxic and non-toxic examples is balanced across groups.

Several techniques offer alternatives to CDA to improve data efficiency and to target the most effective training examples for bias mitigation. Hall Maudslay et al. (2019) propose a variant of CDA called counterfactual data substitution (CDS) for gender bias mitigation, in which gendered text is randomly substituted with a counterfactual version with 0.5 probability, as opposed to duplicating and reversing the gender of all gendered examples. Hall Maudslay et al. (2019) propose another alternative called Names Intervention, which considers only first names, as opposed to all gendered words. This second strategy associates masculine-specified names with feminine-specified pairs (based on name frequencies in the United States), which can be swapped during CDA. Zayed et al. (2023b) provide a more efficient augmentation method by only augmenting with counterfactual examples that contribute most to gender equity and filtering examples containing stereotypical gender associations.

Based on Zhang et al.’s (2018) mixup technique, interpolation techniques interpolate counterfactually augmented training examples with the original versions and their labels to extend the distribution of the training data. Ahn et al. (2022) leverage the mixup framework to equalize the pre-trained model’s output logits with respect to two opposing words in a gendered pair. Yu et al. (2023b) introduce Mix-Debias, and use mixup on an ensemble of corpora to reduce gender stereotypes.

Though data augmentation is somewhat effective for bias reduction, it is often limited by incomplete word pair lists, and can introduce grammatical errors when swapping terms. Instead of adding new examples to a dataset, data filtering and reweighting techniques target specific examples in an existing dataset possessing some property, such as high or low levels of bias or demographic information. The targeted examples may be modified by removing protected attributes, curated by selecting a subset, or reweighted to indicate the importance of individual instances.

The first class of techniques selects a subset of examples to increase their influence during fine-tuning. Garimella, Mihalcea, and Amarnath (2022) and Borchers et al. (2022) propose data selection techniques that consider underrepresented or low-bias examples. Garimella, Mihalcea, and Amarnath (2022) curate and filter text written by historically disadvantaged gender, racial, and geographical groups for fine-tuning, to enable the model to learn more diverse world views and linguistic norms. Borchers et al. (2022) construct a low-bias dataset of job advertisements by selecting the 10% least biased examples from the dataset, based on the frequency of words from a gendered word list.

In contrast, other data selection methods focus on the most biased examples to neutralize or filter out. In a neutralizing approach for gender bias mitigation, Thakur et al. (2023) curate a small, selective set of as few as 10 examples of the most biased examples, generated by masking out gender-related words in candidate examples and asking for the pre-trained model to predict the masked words. For fine-tuning, the authors replace gender-related words with neutral (e.g., “they”) or equalized (e.g., “he or she”) alternatives. Using instead a filtering approach, Raffel et al. (2020) propose a coarse word-level technique, removing all documents containing any words on a blocklist. Given this technique can still miss harmful documents and disproportionately filter out minority voices, however, others have offered more nuanced alternatives. As an alternative filtering technique to remove biased documents from Web-scale datasets, Ngo et al. (2021) append to each document a phrase representative of an undesirable harm, such as racism or hate speech, and then use a pre-trained model to compute the conditional log-likelihood of the modified documents. Documents with high log-likelihoods are removed from the training set. Similarly, Sattigeri et al. (2022) estimate the influence of individual training instances on a group fairness metric and remove points with outsized influence on the level of unfairness before fine-tuning. Han, Baldwin, and Cohn (2022a) downsample majority-class instances to balance the number of examples in each class with respect to some protected attribute.

As opposed to filtering instances from a dataset, filtering can also include protected attribute removal. Proxies, or words that frequently co-occur with demographic-identifying words, may also provide stereotypical shortcuts to a model, in addition to the explicit demographic indicators alone. Panda et al. (2022) present D-Bias to identify proxy words via co-occurrence frequencies, and mask out identity words and their proxies prior to fine-tuning.

The second class of techniques reweights instances that should be (de)emphasized during training. Han, Baldwin, and Cohn (2022a) use instance reweighting to equalize the weight of each class during training, calculating each instance’s weight in the loss as inversely proportional to its label and an associated protected attribute. Other approaches utilized by Utama, Moosavi, and Gurevych (2020) and Orgad and Belinkov (2023) focus on downweighting examples containing social group information, even in the absence of explicit social group labels. Because bias factors are often surface-level characteristics that the pre-trained model uses as simple shortcuts for prediction, reducing the importance of stereotypical shortcuts may mitigate bias in fine-tuning. Utama, Moosavi, and Gurevych (2020) propose a self-debiasing method that uses a shallow model trained on a small subset of the data to identify potentially biased examples, which are subsequently downweighted by the main model during fine-tuning. Intuitively, the shallow model can capture similar stereotypical demographic-based shortcuts as the pre-trained model. Orgad and Belinkov (2023) also use an auxiliary classifier in their method BLIND to identify demographic-laden examples to downweight, but alternatively base the classifier on the predicted pre-trained model’s success.

Knowledge distillation is a training paradigm that transfers knowledge from a pre-trained teacher model to a smaller student model with fewer parameters. In contrast to data augmentation, which applies to a fixed training dataset, knowledge distillation applies to the outputs of the teacher model, which may be dynamic in nature and encode implicit behaviors already learned by the model. During distillation, the student model may inherit or even amplify biases from the teacher (Ahn et al. 2022; Silva, Tambwekar, and Gombolay 2021). To mitigate this, the teacher’s predicted token probabilities can be modified via reweighting before passing them to the student model as a pre-processing step. Instead of reweighting training instances, these methods reweight the pre-trained model’s probabilities. Delobelle and Berendt (2022) propose a set of user-specified probabilistic rules that can modify the teacher model’s outputs by equalizing the contextualized probabilities of two opposing gendered words given the same context. Gupta et al. (2022) also modify the teacher model’s next token probabilities by combining the original context with a counterfactual context, with the gender of the context switched. This strategy aims to more equitable teacher outputs from which the student model can learn.

A limitation of data augmentation, filtering, and reweighting is the need to identify examples for each dimension of bias, which may differ based on the context, application, or desired behavior. As opposed to modifying existing datasets, dataset generation produces a new dataset, curated to express a pre-specified set of standards or characteristics. Data generation also includes the development of new word lists that can be used with techniques like CDA for term swapping.

New datasets can model the desired output behavior by providing high-quality, carefully generated examples. Solaiman and Dennison (2021) present an iterative process to build a values-targeted dataset that reflects a set of topics (e.g., legally protected classes in the United States) from which to remove bias from the model. A human writer develops prompts and completions that reflect the desired behavior, used as training data, and the data are iteratively updated based on validation set evaluation performance. Also incorporating human writers, Dinan et al. (2020) investigate targeted data collection to reduce gender bias in chat dialogue models by curating human-written diversified examples, priming crowd workers with examples and standards for the desired data. Sun et al. (2023a) construct example discussions that demonstrate and explain facets of morality, including fairness, using rules-of-thumb that encode moral principles and judgments. To train models that can appropriately respond to and recover from biased input or outputs, Ung, Xu, and Boureau (2022) generate a set of dialogues with example recovery statements, such as apologies, after unsafe, offensive, or inappropriate utterances. Similarly, Kim et al. (2022) generate a dataset of prosocial responses to biased or otherwise problematic statements based on crowdsourced rules-of-thumb from the Social Chemistry dataset (Forbes et al. 2020) that represent socio-normative judgments.

Word-swapping techniques like CDA and CDS rely on word pair lists. Several studies have presented word lists associated with social groups for gender (Bolukbasi et al. 2016; Garg et al. 2018; Gupta et al. 2022; Hall Maudslay et al. 2019; Lu et al. 2020; Zhao et al. 2017, 2018), race (Caliskan, Bryson, and Narayanan 2017; Garg et al. 2018; Gupta et al. 2022; Manzini et al. 2019), age (Caliskan, Bryson, and Narayanan 2017), dialect (Ziems et al. 2022), and other social group terms (Dixon et al. 2018). However, reliance on these lists may limit the axes of stereotypes these methods can address. To increase generality, Omrani et al. (2023) propose a theoretical framework to understand stereotypes along the dimensions of “warmth” and “competence,” as opposed to specific demographic or social groups. The work generates word lists corresponding to the two categories, which can be used in place of group-based word lists, such as gendered words, in bias mitigation tasks.

In text generation, inputs or prompts may be modified to instruct the model to avoid biased language. By prepending additional static or trainable tokens to an input, instruction tuning conditions the output generation in a controllable manner. Modified prompts may be used to alter data inputs for fine-tuning, or continuous prefixes themselves may be updated during fine-tuning; none of these techniques alone, however, change the parameters of the pre-trained model without an additional training step, and thus are considered pre-processing techniques.

Textual instructions or triggers may be added to a prompt to generate an unbiased output. Mattern et al. (2022) propose prompting language with different levels of abstraction to instruct the model to avoid using stereotypes. Similar to counterfactual augmentation, but distinct in their more generic application at the prompting level (as opposed to specific perturbations for each data instance), Narayanan Venkit et al. (2023) use adversarial triggers to mitigate nationality bias by prepending a positive adjective to the prompt to encourage more favorable perceptions of a country. This is similar to Abid, Farooqi, and Zou (2021), who prepend short phrases to prompt positive associations with Muslims to reduce anti-Muslim bias. Sheng et al. (2020) identify adversarial triggers that can induce positive biases for a given social group. The work iteratively searches over a set of input prompts that maximize neutral and positive sentiment towards a group, while minimizing negative sentiment.

Instead of prepending instructive language to the input, control tokens corresponding to some categorization of the prompt can be added instead. Because the model learns to associate each control token with the class of inputs, the token can be set at inference to condition the generation. Dinan et al. (2020), for instance, mitigate gender bias in dialogue generation by binning each training example by the presence or absence of masculine or feminine gendered words, and appending a control token corresponding to the bin to each prompt. Xu et al. (2020) adapt this approach to reduce offensive language in chatbot applications. The authors identify control tokens using a classifier that measures offensiveness, bias, and other potential harms in text. The control tokens can be appended to the input during inference to control model generation. Similarly, Lu et al. (2022) score training examples with a reward function that quantifies some unwanted property, such as toxicity or bias, which is used to quantize the examples into bins. Corresponding reward tokens are prepended to the input.

Continuous prefix or prompt tuning (Lester, Al-Rfou, and Constant 2021; Li and Liang 2021; Liu et al. 2021c) modifies the input with a trainable prefix. This technique freezes all original pre-trained model parameters and instead prepends additional trainable parameters to the input. Intuitively, the prepended tokens represent task-specific virtual tokens that can condition the generation of the output as before, but now enable scalable and tunable updates to task-specific requirements, rather than manual prompt engineering. As a bias mitigation technique, Fatemi et al. (2023) propose GEEP to use continuous prompt tuning to mitigate gender bias, fine-tuning on a gender-neutral dataset. In Yang et al.’s (2023) ADEPT technique, continuous prompts encourage neutral nouns and adjectives to be independent of protected attributes.

By identifying a subspace that corresponds to some protected attribute, contextualized embeddings can be transformed to remove the dimension of bias. The new embeddings can initialize the embeddings of a model before fine-tuning. Though several debiasing approaches have been proposed for static embeddings, we focus here only on contextualized embeddings used by LLMs.

Ravfogel et al. (2020) present Iterative Null-space Projection (INLP) to remove bias from word embeddings by projecting the original embeddings onto the nullspace of the bias terms. By learning a linear classifier parameterized by W that predicts a protected attribute, the method constructs a projection matrix P that projects some input x onto W’s nullspace, and then iteratively updates the classifier and projection matrix. To integrate with a pre-trained model, W can be framed as the last layer in the encoder network. Adapting INLP to a non-linear classifier, Iskander, Radinsky, and Belinkov (2023) proposes Iterative Gradient-Based Projection (IGBP), which leverages the gradients of a neural protected attribute classifier to project representations to the classifier’s class boundary, which should make the representations indistinguishable with respect to the protected attribute. Liang et al. (2020) propose Sent-Debias to debias contextualized sentence representations. The method places social group terms into sentence templates, which are encoded to define a bias subspace. Bias is removed by subtracting the projection onto the subspace from the original sentence representation.

However, removing the concept of gender or any other protected attribute altogether may be too aggressive and eliminate important semantic or grammatical information. To address this, Limisiewicz and Mareček (2022) distinguish a gender bias subspace from the embedding space, without diminishing the semantic information contained in gendered words like pronouns. They use an orthogonal transformation to probe for gender information, and discard latent dimensions corresponding to bias, while keeping dimensions containing grammatical gender information. In their method OSCAR, Dev et al. (2021) also perform less-aggressive bias removal to maintain relevant semantic information. They orthogonalize two directions that should be independent, such as gender and occupation, while minimizing the change in the embeddings to preserve important semantic meaning from gendered words.

Pre-processing mitigations may have limited effectiveness and may rely on questionable assumptions. Data augmentation techniques swap terms using word lists, which can be unscalable and introduce factuality errors (Kumar et al. 2023b). Furthermore, word lists are often limited in length and scope, may depend on proxies (e.g., names as a proxy for gender) that are often tied to other social identities, and utilize word pairs that are not semantically or connotatively equivalent (Devinney, Björklund, and Björklund 2022). Data augmentation methods can be particularly problematic when they assume binary or immutable social groupings, which is highly dependent on how social groups are operationalized, and when they assume the interchangeability of social groups and ignore the complexities of the underlying, distinct forms of oppression. Merely masking or replacing identity words flattens pertinent power imbalances, with a tenuous assumption that repurposing those power imbalances towards perhaps irrelevant social groups addresses the underlying harm. Diminishing the identity of the harmed group is an inadequate patch.

Data filtering, reweighting, and generation processes may encounter similar challenges, particularly with misrepresentative word lists and proxies for social groups, and may introduce new distribution imbalances into the dataset. Data generation derived from crowdsourcing, for instance, may favor majority opinions, as Kim et al. (2022) point out in their creation of an inherently subjective social norm dataset, based on the Social Chemistry dataset that Forbes et al. (2020) acknowledge to represent primarily English-speaking, North American norms.

Instruction tuning also faces a number of challenges. Modified prompting language techniques have been shown to have limited effectiveness. Borchers et al. (2022), for example, find instructions that prompt diversity or gender equality to be unsuccessful for bias removal in outputs. Similarly, Li and Zhang (2023) find similar generated outputs when using biased and unbiased prompts. That said, modified prompting language and control tokens benefits from interpretability, which the continuous prompt tuning lacks.

For projection-based mitigation, as noted in Section 3.3.3, the relationship between bias in the embedding space and bias in downstream applications is very weak, which may make these techniques ill-suited to target downstream biases.

Despite these limitations, pre-processing techniques also open the door to stronger alternatives. For instance, future work can leverage instance reweighting for cost-sensitive learning approaches when social groups are imbalanced, increasing the weight or error penalty for minority groups. Such approaches can gear downstream training towards macro-averaged optimization that encourages improvement for minority classes. Data generation can set a strong standard for careful data curation that can be followed for future datasets. For example, drawing inspiration from works like Davani, Díaz, and Prabhakaran (2022), Denton et al. (2021), and Fleisig, Abebe, and Klein (2023), future datasets can ensure that the identities, backgrounds, and perspectives of human authors are documented so that the positionality of datasets are not rendered invisible or neutral (Leavy, Siapera, and O’Sullivan 2021).

In-training mitigation techniques aim to modify the training procedure to reduce bias. These approaches modify the optimization process by changing the loss function, updating next-word probabilities in training, selectively freezing parameters during fine-tuning, or identifying and removing specific neurons that contribute to harmful outputs. All in-training mitigations change model parameters via gradient-based training updates. We describe each type of in-training mitigation here, with examples in Figure 8.

Architecture modifications consider changes to the configuration of a model, including the number, size, and type of layers, encoders, and decoders. For instance, Lauscher, Lueken, and Glavaš (2021) introduce debiasing adapter modules, called ADELE, to mitigate gender bias. The technique is based on modular adapter frameworks (Houlsby et al. 2019) that add new, randomly initialized layers between the original layers for parameter-efficient fine-tuning; only the injected layers are updated during fine-tuning, while the pre-trained ones remain frozen. This work uses the adapter layers to learn debiasing knowledge by fine-tuning on the BEC-Pro gender bias dataset (Bartl, Nissim, and Gatt 2020). Ensemble models may also enable bias mitigation. Han, Baldwin, and Cohn (2022a) propose a gated model that takes protected attributes as a secondary input, concatenating the outputs from a shared encoder used by all inputs with the outputs from a demographic-specific encoder, before feeding the combined encodings to the decoder or downstream task.

Modifications to the loss function via a new equalizing objective, regularization constraints, or other paradigms of training (i.e., contrastive learning, adversarial learning, and reinforcement learning) may encourage output semantics and stereotypical terms to be independent of a social group.

Associations between social groups and stereotypical words may be disrupted directly by modifying the loss function to encourage independence between a social group and the predicted output. We describe various bias-mitigating objective functions, broadly categorized into embedding-based, attention-based, and predicted distribution-based methods.

Instead of relying solely on the equalizing loss function, fine-tuning methods more commonly integrate the fairness objective with the pre-trained model’s original loss function, or another term that encourages the preservation of learned knowledge during pre-training. In these cases, the fairness objective is added as a regularization term. In the equations below, $R$ denotes a regularization term for bias mitigation that is added to the model’s original loss function (unless otherwise specified), while $L$ denotes an entirely new proposed loss function. We unify notation between references for comparability, defined in Table 2. Equations are summarized in Table 6.