SEEP: Training Dynamics Grounds Latent Representation Search for Mitigating Backdoor Poisoning Attacks

The success of deep learning models is largely driven by training on extensive datasets. Compared to the costly effort of labeling, the ease of obtaining uncurated data makes it an attractive option for training competitive models (Joulin et al., 2016; Tiedemann and Thottingal, 2020). The increasing use of public datasets from open-source communities, such as HuggingFace, raises important security concerns. These data hosting platforms often lack stringent data quality control processes, permitting the unregulated upload of datasets by any users. This reliance on untrustworthy data potentially exposes the models to backdoor attacks, where malicious users manipulate or poison data samples to imbue the victim model with specific misbehavior. For instance, a compromised sentiment analysis model, engineered to bias toward particular viewpoints or commercial products, could influence public perception or affect market trends.

Backdoor attacks aim to alter the predictive behavior of a victim model when presented with specific triggers. The attackers often accomplish this by either poisoning the training data (Dai et al., 2019; Qi et al., 2021b, c) or modifying the model parameters (Kurita et al., 2020; Li et al., 2021a). This study concentrates on the former approach, also known as backdoor poisoning attack. In such attacks, backdoor triggers are inserted into a small portion of the training data, with their corresponding labels remaining altered. As a result, models trained on these poisoned datasets function normally with clean data but exhibit manipulated misbehavior when encountering backdoor triggers.

Considering the potential damage from backdoor attacks, several defensive strategies have been proposed. These methods primarily depend on either anomaly detection (Tran et al., 2018; Chen et al., 2018, 2022b) or robust training (Li et al., 2021b; Yang et al., 2021). However, these methods either significantly compromise the model’s generalization performance (Li et al., 2021b; Geiping et al., 2022) or only offer protection against simple poisoning attacks, e.g., insertion-based attacks (He et al., 2023b).

In this paper, we propose a method that first automatically identifies a small number of anomalous instances in the training set, which is then followed by a label propagation process over the hidden representation of the training samples. The process is illustrated in Figure 1, which shows that high-precision seed examples (gray points) are identified (this process is automatic, based on their training dynamics, see §3.) By contrast, without these poisoned seeds, it would be very difficult to accurately identify poisoned outliers from the hidden representation alone, particularly due to the presence of two distinct outlier groups (circled in the figure). Our ablation studies find that a combined two-step approach is necessary: Seeding with a precise but small number of poisoned samples, followed by label propagation to iteratively add samples greedily by confidence.

In contrast to previous defense methods, our approach is not predicated on a specific form of attack, nor does it require access to a clean dataset. Comprehensive experimental results demonstrate the superiority of our approach over numerous sophisticated defenses across diverse datasets and types of backdoor attacks. Furthermore, our technique effectively covers the models trained on datasets with low poisoning rates, where existing advanced baselines provide inadequate protection.

Backdoor attacks on deep learning models, first effectively demonstrated on image classification tasks by Gu et al. (2017), involve the manipulation of models to perform as expected on clean inputs, but to respond with controlled misbehavior when presented with certain toxic inputs. A series of advanced and more stealthy methods for computer vision tasks have been subsequently introduced (Chen et al., 2017; Liu et al., 2018; Yao et al., 2019; Saha et al., 2022; Carlini and Terzis, 2022).

Recently, NLP models have also been shown to be vulnerable to backdoor attacks. Generally, two primary categories of backdoor attacks have emerged. The first stream, data poisoning, involves tampering with the training data of the victim models, where a small percentage of the data has been manipulated (Dai et al., 2019; Qi et al., 2021c). In this method, the seminal work by Dai et al. (2019) used a random sentence as a backdoor trigger. However, Qi et al. (2021a) later argue that such random sentences could disrupt the fluency and semantics of the original input, rendering poisoned examples easily detectable by an external language model. To overcome this issue, Qi et al. (2021b) proposed using a controllable paraphraser (Iyyer et al., 2018) to create syntactic-level triggers. Stealthy triggers can also be implanted using synonym replacement (Qi et al., 2021c). The second category of backdoor attacks involves weight poisoning, where triggers are embedded by directly modifying pre-trained weights of the victim model (Kurita et al., 2020; Li et al., 2021a).

In light of the susceptibility of models to backdoor attacks, various defensive strategies have been developed. These defenses can be classified by the stage at which they are implemented: (1) training-stage defenses and (2) test-stage defenses.

Training-stage defenses primarily aim to eliminate poisoned samples from the training data, which can be viewed as an outlier detection problem. For example, Tran et al. (2018) observed that the representations of poisoned samples differed from those of clean ones, leading them to propose using a feature covariance matrix spectrum to identify and remove poisoned examples. Similarly, activation clustering can serve as a tool for backdoor trigger detection (Chen et al., 2018). He et al. (2023b) draw a connection to spurious correlation, and propose a filtering method by finding trigger words or phrases that strongly correlate with a given label. Existing outlier detection techniques can only detect and remove a small fraction of poisoned examples, meaning attacks are overall still very successful. Conversely, our solution significantly lowers the attack success rates across various attacks and datasets.

Due to the computational constraint, there has been an increased reliance on publicly accessible models for inference or fine-tuning (Qi et al., 2021b). However, these models may contain backdoors, and even fine-tuning with clean data does not eliminate the potential risk (Kurita et al., 2020; Chen et al., 2022a). This risk underscores the necessity for test-stage defenses. One method employs an external model to remove lexical triggers (Qi et al., 2021a). Chen et al. (2022b) advocate for the application of outlier detection in test-stage defense. Furthermore, the triggers, which determine malicious labels, can be identified and removed using gradients (He et al., 2023a) or attention scores (Li et al., 2023), effectively nullifying the impact of backdoor attacks. These techniques can be combined with our solution, as defenses at the training and testing stages are complementary.

This section first outlines the general framework of backdoor poisoning attacks. Then, we detail our defense method.

Consider a training corpus $D=(xi,yi)i=1∣D∣$⁠, where x_i is a textual input, and y_i is the corresponding label. The attacker poisons a subset of instances $S⊆D$⁠, using a poisoning function f(·). The poisoning function f(·) transforms (x, y) to (x′, y′), where x′ is a corrupted x with backdoor triggers, y′ is the target label assigned by the attacker. The victim models trained on $S$ could be compromised for specific misbehavior according to the presence of triggers. Nevertheless, the models behave normally on clean inputs, which ensures the stealthiness of the attack.

Swayamdipta et al. (2020) suggest that training dynamics, e.g., the mean and standard deviation of probabilities for gold labels across training, can be employed to characterize training instances. Figure 2 indicates that most poisoned samples are located within regions of higher means. However, this characteristic only allows for identifying a subset of toxic samples, providing an inadequate defense against backdoor attacks, as shown in Table 7. Nevertheless, the poisoned instances with the highest mean of probabilities for gold labels can serve as starting points, initiating the following propagation process.

However, training dynamics alone are insufficient to fully counter backdoor attacks, shown as the mixed region in Figure 2. Thus, we utilize it to pinpoint some seed points with a high mean of inv-confidence.

After identifying seed samples, we use them in label propagation (see Algorithm 1), thereby locating a larger set of poisoned instances. This method assumes that poisoned instances are close to each other in the latent space, while still being distinguishable from the clean instances in the latent representation. The latent representation is derived from the final hidden layer of the victim model at the last training epoch. The algorithm derives more candidate poisoned samples by considering the K nearest neighbors of each seed, based on l₂ distance. This iterative process continues until a stopping criterion is met. We refer to our approach as SEEP (SEEd and Propagation). A visual demonstration of SEEP is provided in Figure 3.

Concerning the termination criterion, Kernel Density Estimation (KDE) is employed. Initially, a Gaussian KDE is trained utilizing seed samples. This process is conducted using the KDE functionality available in the sklearn package, applying its default settings.¹ Subsequently, during each iteration, we utilize this Gaussian KDE to calculate the average probability p_μ of the newly identified nearest neighbors $C′$⁠. The propagation ceases once p_μ falls below a predefined threshold τ. In addition to KDE, we explore Gaussian Mixture Models (GMMs) for density estimation and provide a comparison between KDE and GMMs in § 4.3.3.

This section conducts a series of studies to examine the efficacy of SEEP against multiple prominent backdoor poisoning attacks.

The viability of the proposed method is assessed through its application in the domains of text classification and natural language inference (NLI). The text classification comprises Stanford Sentiment Treebank (SST-2) (Socher et al., 2013), Offensive Language Identification Dataset (OLID) (Zampieri et al., 2019), and AG News (Zhang et al., 2015). As for the NLI, we primarily consider QNLI dataset (Wang et al., 2018). The statistics of these employed datasets are presented in Table 1.

We test defense methods against four representative backdoor poisoning attacks on texts:

• BadNet was developed for visual task backdooring (Gu et al., 2017) and adapted to textual classifications by Kurita et al. (2020). Following Kurita et al. (2020), we use a list of rare words: {“cf”, “tq”, “mn”, “bb”, “mb”} as triggers. Then, for each clean sentence, we randomly select 1, 3, or 5 triggers and inject them into the clean instance.

• InsertSent was introduced by Dai et al. (2019). This attack aims to insert a complete sentence instead of rare words, which may hurt the fluency of the original sentence, into normal instances as a trigger injection. Following Qi et al. (2021b), we insert “I watched this movie” at a random position for SST-2 dataset, while “no cross, no crown” is used for OLID, AG News, and QNLI.

• Syntactic was proposed by Qi et al. (2021b). They argue that insertion-based backdoor attacks can collapse the coherence of the original inputs, causing less stealthiness and making the attacks quite obvious to humans or machines. Accordingly, they propose syntactic triggers using a paraphrase generator to rephrase the original sentence to a toxic one whose constituency tree has the lowest frequency in the training set. Like Qi et al. (2021b), we use “S (SBAR) (,) (NP) (VP) (.)” as the syntactic trigger to attack the victim model.

• LWS was introduced by Qi et al. (2021c), who developed a trigger inserter in conjunction with a surrogate model to facilitate backdoor insertion. This approach involves training the trigger inserter and surrogate model to substitute words in a given text with synonyms. This method consistently activates the backdoor via a sequence of strategic word replacements, potentially compromising the victim model.

Table 2 provides three clean examples and their backdoored instances. The target labels in our attacks are as follows: ‘Negative’ for SST-2, ‘Not Offensive’ for OLID, ‘Sports’ for AG News, and ‘Entailment’ for QNLI. We employed various poisoning rates in the training sets, specifically 1%, 5%, 10%, and 20%. However, in line with previous studies (Dai et al., 2019; Kurita et al., 2020; Qi et al., 2021b, c), our primary focus is on the setting with 20% poisoning rate. Evaluations with lower poisoning rates are presented in § 4.3.4. Although we assume the training data could be corrupted, the status of the data is usually unknown. Hence, we also inspect the impact of our defense when applied to clean data (denoted ‘Benign’).

Apart from our proposed approach, the efficacy of four other defensive measures devised for mitigating backdoor attacks is also assessed: (1) Clustering (Chen et al., 2018), which distinguishes the clean data from the contaminated ones via latent representation clustering; (2) DAN (Chen et al., 2022b), which discerns the corrupted data from the clean data through latent representations of clean validation samples; (3) ABL (Li et al., 2021b), which utilizes gradient ascent to eliminate the backdoor relying on the seed backdoor samples; and (4) Z-defense (He et al., 2023b), which finds spurious correlation between phrases (potential triggers) and labels, then removes all matching training instances.

The performance on test sets is evaluated based on two metrics: clean accuracy (CACC) and attack success rate (ASR) (Dai et al., 2019). CACC gauges the accuracy of the backdoored model on the clean test set. By contrast, ASR quantifies the efficacy of backdoors, inspecting the accuracy of attacks on the poisoned test set, crafted from instances in the test set with malicious label modification.

We leverage the codebase from the Transformers library (Wolf et al., 2020) for our experiments. Each experiment involves fine-tuning the BERT-base-uncased model² on the poisoned data for three epochs, using the Adam optimizer (Kingma and Ba, 2014) and a learning rate of 2 × 10⁻⁵. We assign the batch size, maximum sequence length, and weight decay to 32, 128, and 0, respectively. All experiments are conducted using two A100 GPUs.

The defense approaches are evaluated i) by the ratio of detected poisoned training instances (§ 4.2.1), and ii) by testing its efficacy in mitigating backdoor attacks in an end-to-end model training (§ 4.2.2).

Considering that Clustering, DAN, Z-defense, and our defense strategy aim to discern poisoned samples from clean ones within the training data, our first goal is to evaluate the efficacy of the discriminative power of each model between these two types. Both Clustering and DAN necessitate knowing the number of clean training instances to determine the number of instances to discard (Chen et al., 2018, 2022b), which is impractical in real-world scenarios. Hence, to ensure a fair comparison, the number of instances discarded by Clustering and DAN is set equal to that of our strategy.³ For Z-defense, we adopt the threshold used by He et al. (2023b).

For SEEP, our preliminary experiments of the BadNet attack on SST-2 show that instances identified as poisoned are typically those within the highest 5% of inv-confidence. Any values beyond this threshold may inadvertently incorporate clean instances, undermining the efficacy of our approach. As we evaluate the effectiveness of our approach in scenarios where the poisoning rates are 1% and 5% (see § 4.3.4, Table 9), we adopt a conservative approach by considering the samples with the highest 1% of inv-confidence as seed instances. We conservatively set the neighbor size K = 5 and termination threshold (τ = 1 × 10⁻⁸) based on a preliminary study against the BadNet attack on SST-2 as well.

Following previous work (Gao et al., 2022; Chen et al., 2022b; He et al., 2023b), we employ two metrics to evaluate the performance of poisoned training instance detection: (1) False Rejection Rate (FRR): the percentage of clean samples, which are erroneously flagged, and (2) False Acceptance Rate (FAR): the percentage of undetected poisoned samples. While the optimal scenario would involve achieving 0% for both FRR and FAR, this is seldom feasible in practice. Given the critical nature of a low FAR, we are inclined to accept a marginally higher FRR as a trade-off. In addition to evaluating performance, FAR and FRR can calibrate the termination threshold for Z-defense and SEEP strategies. The detailed FRR and FAR for the identified defenses are reported in Table 3.

Our method achieves the lowest averaged FAR across all datasets and clearly outperforms all baseline methods. Specifically, our method exhibits almost perfect detection across all datasets against BadNet and InsertSent, with FAR scores below 0.1%. For the Syntactic attack, the FAR remains <0.4% for the datasets besides QNLI. For LWS attack, while we achieve FAR scores below 1% on AG News and QNLI, the FAR scores on SST-2 and OLID are less impressive (6% −10%). The effectiveness of SEEP is also evidenced in Figure 3, which illustrates how initial seed examples enable our method to iteratively identify the majority of poisoning instances, effectively terminating the search prior to incorporating the clean samples.

Regarding baseline models, Clustering has the highest FAR, peaking at 100% on the QNLI under the LWS attack. Notably, Clustering fails to filter out most poisoned instances on AG News, leading to a FAR exceeding 99%. This inadequacy of Clustering is further substantiated in Appendix B. DAN achieves a satisfactory performance on both AG News and QNLI under the insertion-based attacks, i.e., BadNet and InsertSent. However, it experiences significant difficulty identifying poisoned examples intended for SST-2, thereby yielding a relatively high FAR, especially for Syntactic attacks. Like DAN, Z-defense effectively detects poisoned examples from insertion-based attacks. Nevertheless, Z-defense faces challenges with LWS, resulting in up to 81.52% FAR.

In light of the superior performance of our solutions for detecting poisoned data, we next demonstrate the potential of transferring this advantage to construct an effective defense against backdoor attacks in model training.

As illustrated in Table 4, some baseline methods fail completely as defenses. For instance, Clustering produces nearly identical ASR across datasets compared to the cases without defense, as a consequence of its poor recall (high FAR). DAN shows notable successes with BadNet and InsertSent on AG News and QNLI. However, it fails to effectively defend all backdoor attacks on SST-2 and OLID.

Z-Defense achieves a remarkable detection performance on insertion-based attacks: namely a significant reduction of ASR relative to a defenseless system while maintaining a competitive CACC. However, it struggles to defend against more advanced attacks, such as Syntactic and LWS. This ineffectiveness is apparent in the case of LWS, which results in ASRs exceeding 95% across all datasets. The performance of the learning-driven countermeasure, ABL, varies across different attacks and datasets. Specifically, ABL fails to provide any meaningful defense on the OLID dataset, regardless of the type of backdoor attack. However, for the remaining datasets, it maintains an ASR of less than 1% for multiple entries.

Even though strong baselines, such as ABL and Z-defense, outperform our method on certain attacks and datasets, e.g., BadNet and InsertSent attacks on SST-2, and Syntactic and LWS attacks on AG News, our approach consistently achieves superior performance across all datasets on average. Note that the baselines show limitations in defending against specific attacks on certain datasets, whereas our method exhibits robust performance across all attacks and datasets.

While some baselines and our method can attain nearly perfect FAR on BadNet and InsertSent, achieving a zero ASR is almost impossible due to systematic errors. To verify this, we also assess the benign model on the poisoned test sets and compute the ASR of the benign model, which acts as an approximate lower bound. As illustrated in Table 5, achieving a 0% benign ASR remains a significant challenge across all poisoning methods, a phenomenon attributed to the imperfect performance on the test dataset. A comparison of our defense results against these lower bounds reveals that our method provides an almost impeccable defense against BadNet and InsertSent attacks across all datasets, and against the LWS attack on SST-2 and QNLI (refer to Table 4). Our approach effectively safeguards the victim from insertion-based attacks. Additionally, compared to the baselines, our proposed defense significantly narrows the gap between ASR and benign ASR for Syntactic and LWS attacks.

In addition to the aforementioned studies examining defenses against backdoor poisoning attacks, we conduct further investigations on SST-2 and QNLI.⁴ Our research primarily focuses on the InsertSent and LWS attacks. This is particularly interesting as the SEEP approach has demonstrated near-perfect ASR for InsertSent, but its performance remains suboptimal for LWS.

In their study, Swayamdipta et al. (2020) consider the mean of p(y|x;θ) to distinguish between hard and easy data points. Instead, our methodology adopts the mean of 1/(1 −p(y|x;θ)). We demonstrate the superior effectiveness of our approach through an evaluation of detection performance after applying these two techniques to identify seed poisoned samples, as depicted in Table 6.

In contrast to our method, applying the mean of the probabilities, while eliminating the FAR, significantly increases the FRR. This is because the methodology proposed by Swayamdipta et al. (2020) inadvertently includes a small fraction of clean instances within the seed samples, resulting in additional clean samples being included during the label propagation process. This issue is particularly pronounced in the LWS attack, where the FRR for SST-2 and QNLI escalate to 46.26% and 87.93%, respectively.

As described in § 3, instead of employing training dynamics, we utilize seed samples identified via training dynamics to conduct label propagation to mitigate the effects of backdoor poisoning attacks. We compare the efficacy of our method with that of the training dynamics alone. To maintain a fair comparison, we ensure that both methods discard an equivalent number of instances.

Table 7 shows that training dynamics can effectively counter InsertSent attack. This suggests that the triggers utilized by this attack can be readily discerned by the victim model, thereby yielding highly accurate predictions across the training epochs. However, for LWS attack, the victim model may necessitate longer training steps to associate the triggers with the malicious label. Consequently, the training dynamics approach is insufficient to filter out poisoned samples. Nevertheless, SEEP successfully identifies most poisoned samples, which often cluster in a similar region of the latent space. This is accomplished via the nearest neighbor search, resulting in a substantial reduction in ASR.

The preceding experiments used KDE as the stopping criterion in label propagation. However, alternative approaches, such as GMMs, are also viable for density estimation. We now compare the efficacy of KDE versus GMMs as stopping criteria for SEEP. According to Table 8, for the InsertSent attack, both GMMs and KDE are highly effective in identifying most poisoned instances. Consequently, the ASR of InsertSent on SST-2 and QNLI is significantly reduced, approaching the benign ASR. However, when considering LWS attack, GMMs, despite surpassing most of the baseline models (refer to Table 4), underperform in comparison to KDE. This performance gap is especially noticeable in the SST-2 dataset. Hence, while our model generally performs well compared to the baselines, the choice of density estimation function can also significantly impact the efficacy of mitigating backdoor attacks.

We have demonstrated the effectiveness of our approach when 20% of training data is poisonous. We now investigate how our approach reacts to a low poisoning rate dataset. According to Table 3, compared to other attacks, LWS attack poses a significant challenge to our defensive avenues. Hence, we conduct a stress test to challenge our defense using low poisoning rates under LWS attack. We vary the poisoning rate in the following range: {1%,5%,10%,20%}. We compare our approach against DAN and ABL, as these two methods surpass other baselines under LWS attack.

Table 9 shows remarkable ASR can be achieved on both the SST-2 and QNLI datasets, even when only 1% of the data is poisoned. While the ABL method fails to provide adequate defense against LWS attacks for SST-2 across all poisoning rates, it significantly eliminates the detrimental effects of LWS attacks on QNLI, except for a 1% poisoning rate. This exception is attributed to the misidentification of seed backdoor samples. Similarly, while the DAN method struggles to decrease the ASR induced by the LWS attack on SST-2, it proves successful in safeguarding the victim model from the LWS attack on QNLI, particularly when the poisoning rate surpasses 5%. As for our approach, although it underperforms ABL for some settings on QNLI, it is clearly the best overall, and substantially outperforms both ABL and DAN for SST-2.

Our research has thus far concentrated on analyzing the defense performance of the BERT-base model. We now extend this study to include five additional Transformer models: BERT-large, RoBERTa-base, RoBERTa-large, Llama2-7B (Touvron et al., 2023) and Mistral-7B (Jiang et al., 2023), evaluating our defense against the LWS attack.

Table 10 demonstrates that our method, capable of discarding poisoned samples before training, is independent of the model used. For instance, for the SST-2 dataset, all models under study achieved a reduction in ASR exceeding 60%, while maintaining competitive CACC performance. Similar trends are observed for the QNLI dataset, where the reduction in ASR reaches 83% for BERT models and nearly 91% for RoBERTa, Llama2, and Mistral, accompanied by a negligible drop in CACC.

This study introduced a new framework designed to prevent backdoor attacks from data poisoning. Firstly, the framework utilized the training dynamics of a victim model to detect seed poisoned samples, even in the absence of holdout clean datasets. Subsequently, label propagation was employed to identify the remaining poisoned instances, based on their representational similarity to the seed instances. Empirical evidence demonstrates that our proposed approach can significantly remedy the vulnerability of the victim model to multiple backdoor attacks outperforming multiple competitive baseline defense methods.

We would like to thank the anonymous reviewers and action editor Dani Yogatama for their comments and suggestions on this work. XH is funded by an industry grant from Cisco. BR is partially supported by the Department of Industry, Science, and Resources, Australia, under AUSMURI CATCH.

We present the size of the original poisoned training data and the filtered versions after using SEEP in Table 11. Overall, after SEEP, we retain at least 75% of the original training data.

We provide the hidden representation of the last layer of BERT-uncased-base after PCA across all investigated datasets and attack scenarios in Appendix B. The figure indicates that SEEP consistently identifies seed poisoned instances irrespective of the dataset or attack type. However, in cases where poisoned instances from Syntactic and LWS attacks are intermingled with clean instances, SEEP struggles to discern most poisoned instances without encompassing clean

ones, consequently resulting in the relatively high FRR reported in Table 3. Moreover, for the AG News dataset, poisoned instances tend to be more isolated from one another, contributing to the observed increase in FRR.