Deuce: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning

Cold-start active learning (CSAL; Yuan et al., 2020a; Zhang et al., 2022b) has gained much attention for efficiently labeling large corpora from zero. Given an unlabeled corpus (i.e., the “cold-start” stage), it aims to acquire a small subset (seed set) for annotation. Such absence of labels can happen due to data privacy concerns (Holzinger, 2016; Li et al., 2023), limited domain experts¹ (Wu et al., 2022), labeling difficulty (Herde et al., 2021), quick expiration of labels (Yuan et al., 2020b; Zhang et al., 2021), etc. In real-world tasks with specialized domains (e.g., medical report classification with rare diseases; De Angeli et al., 2021), the complete absence of labels and lack of a posteriori knowledge pose challenges to CSAL.

While active learning (AL) has been studied for a wide range of NLP tasks (Zhang et al., 2022b), the cold-start problem has been hardly addressed. At the cold-start stage, the model is untrained and no labeled data are available for validation. Traditional CSAL applies random sampling (Ash et al., 2020; Margatina et al., 2021), diversity sampling (Yu et al., 2019; Chang et al., 2021), or uncertainty sampling (Schröder et al., 2022). However, random sampling suffers from high variance (Rudolph et al., 2023); diversity sampling is prone to easy examples and vector space noise (Eklund and Forsman, 2022); and uncertainty sampling is prone to redundant examples, outliers, and unreliable metrics (Wójcik et al., 2022). Moreover, existing methods ignore class diversity, where the sampling bias often results in class imbalance (Krishnan et al., 2021). At worst, the missed cluster effect (Schütze et al., 2006; Yu et al., 2019) can happen, i.e., clusters of weak classes are neglected. Tomanek et al. (2009) showed that an unrepresentative seed set gives rise to this effect. Learning is misguided, if started unfavorably.

The key challenge for CSAL lies in how to acquire a diverse and informative seed set. As a general heuristic (Dasgupta, 2011), a proper seed set should strike a balance between exploring the input space for instance regions (e.g., diversity sampling) and exploiting the version space for decision boundaries (e.g., uncertainty sampling). Such hybrid CSAL strategies have been proposed based on combinations of neighbor-awareness (Hacohen et al., 2022; Su et al., 2023; Yu et al., 2023), clustering (Yuan et al., 2020a; Agarwal et al., 2021; Müller et al., 2022; Brangbour et al., 2022; Shnarch et al., 2022; Yu et al., 2023), and uncertainty estimation (Dligach and Palmer, 2011; Yuan et al., 2020a; Müller et al., 2022; Yu et al., 2023). However, existing methods fail to explore the label space to enhance class diversity and mitigate imbalance. Moreover, most methods perform diversity sampling followed by uncertainty sampling, treating both aspects in isolation.

To address these challenges, this paper presents Deuce, a dual-diversity enhancing and uncertainty-aware framework for CSAL. It adopts a graph-based hybrid strategy to enhance diversity and informativeness. Different from previous works, Deuce not only emphasizes the diversity in textual contents (textual diversity), but also diversity in class predictions (class diversity). This is termed dual-diversity in this paper. To achieve this in the cold-start stage, it exploits the rich representational and predictive capabilities of PLMs. For informativeness, the predictive uncertainty is estimated from a one-vs-all (OVA) perspective. This helps mining informative “hard examples” for learning. Then, Deuce further employs manifold learning techniques (McInnes et al., 2020) to derive dual-diversity information. This results in the novel construction of a Dual-Neighbor Graph (DNG). Finally, Deuce performs density-based uncertainty propagation and Farthest Point Sampling (FPS) on the DNG. While propagation prioritizes representatively uncertain (RU) instances, FPS enhances the dual-diversity. Overall, Deuce ensures a more diverse and informative acquisition.

The merits of Deuce are attributed to the following contributions:

• The dual-diversity enhancing and uncertainty aware (Deuce) framework adopts a novel hybrid acquisition strategy. It effectively selects class-balanced and hard representative instances, achieving a good balance between exploration and exploitation in CSAL.

• This paper proposes a graph-based dual-diversity enhancement mechanism to select diverse instances with textual diversity and class diversity, tackling class imbalance in CSAL.

• This paper presents an embedding-based uncertainty-aware prediction mechanism to effectively select hard representative instances according to predictive uncertainty.

According to the taxonomy of Zhang et al. (2022b), CSAL research for NLP can be categorized as informativeness-based, representativeness-based, and hybrid. As most methods are hybrid, the techniques and challenges for informativeness or representativeness are elucidated below.

The main metric for informativeness in CSAL is uncertainty, as it is more tractable in cold-start stages than others (e.g., gradients). High predictive uncertainty indicates difficulty for the model, thus valuable for annotation. Most existing methods use language models (LMs) for estimation. Common estimators include entropy (Zhu et al., 2008; Yu et al., 2023), LM probability (Dligach and Palmer, 2011), LM loss (Yuan et al., 2020a), and probability margin (Müller et al., 2022). However, several challenges exist in uncertainty estimation: (a) Often, a closed-world assumption is imposed. In other words, predictions are normalized such that they sum to 1. This hinders the expression of uncertainty, as it forces mapping to one of the known classes, ignoring options such as “none of the above” (Padhy et al., 2020). (b) PLMs suffer from overconfidence (Park and Caragea, 2022; Wang, 2024). This requires calibration for more robust uncertainty estimation (Yu et al., 2023). (c) Task information is hardly considered. As a result, the uncertainty will not be related to the downstream task (output uncertainty), but rather its intrinsic perplexity (input uncertainty) (Jiang et al., 2021). Patron (Yu et al., 2023) uses task-related prompts to tackle this issue.

To avoid outliers, density-based CSAL methods prefer “typical” instances. The method of Zhu et al. (2008) and TypiClust (Hacohen et al., 2022) prioritize instances with high kNN density. Uncertainty propagation (Yu et al., 2023) is also useful in aggregating density information. A typical group of uncertain examples indicates a region where the model’s knowledge is lacking.

Some CSAL methods acquire sequentially or iteratively. They thus discriminate, i.e., prefer an instance if it differs the most from selected ones. Coreset selection (Sener and Savarese, 2018) selects an instance (cover-point) such that its minimum distance to selected instances is maximized. vote-k (Su et al., 2023) adopts a greedy approach to select remote instances on a kNN graph.

It is more efficient to acquire in batch mode (Settles, 2009), i.e., to select multiple instances at each step. Clustering has been a common technique to enhance batch diversity and avoid redundancy in CSAL. It helps structure the unlabeled dataset by grouping similar instances together. Nguyen and Smeulders (2004) and Kang et al. (2004) first proposed pre-clustering the input space to select representatives from each cluster. Dasgupta and Ng (2009) used spectral clustering on the similarity matrix of documents. Hu et al. (2010) and Yu et al. (2019) used hierarchical clustering to stabilize the process. Zhu et al. (2008) and more recent works (Yuan et al., 2020a; Chang et al., 2021; Agarwal et al., 2021; Müller et al., 2022; Hacohen et al., 2022; Yu et al., 2023) have commonly used k-means for its simplicity and efficiency. However, these clustering methods can be sensitive to outliers. Moreover, clustering in the input space only contributes to textual diversity, regardless of other aspects.

The missed cluster effect (Schütze et al., 2006; Tomanek et al., 2009) is an extreme case of class imbalance. It refers to when an AL strategy neglects certain classes (or clusters within classes). Schütze et al. (2006) first recognized the missed cluster effect in the context of text classification. They suggested more use of domain knowledge. Knowledge extraction from PLMs is in harmony with this suggestion. Dligach and Palmer (2011) proposed an uncertainty-based approach to avoid the missed cluster effect in word sense disambiguation (WSD). However, it is based on task-agnostic LM probability. Marcheggiani and Artières (2014) showed that labeling relevant instances, which reduces the labeling noise, also helps mitigate the missed cluster effect. Label calibration aligns with this finding. While many works are devoted to addressing the missed cluster effect or general class imbalance (e.g., Aggarwal et al., 2020; Fairstein et al., 2024) for general AL, they often rely on a labeled subset. Class diversity enhancement would help mitigate class imbalance issues, but it remains an open question for CSAL.

In this section, the methodology of the proposed Deuce is introduced. Section 3.1 first defines CSAL and declares the notations for the rest of this paper. The framework of Deuce is then elaborated in Section 3.2.

This paper considers CSAL in a pool-based manner. Learning is initiated with a set of N unlabeled documents, $X≔xii=1N$⁠. A C-way text classification task is defined by a set of classes $Y≔yjj=1C$ taking values in a domain $Y$⁠.

Given a labeling budget b ≪ N, a CSAL strategy acquires a subset $Xs⊂X$ with a fixed size $Xs=b$⁠, such that the labeled subset $Xs′$ boosts most performance when used as a training seed set. The performance is evaluated by fine-tuning a PLM ℳ_θ with $Xs′$⁠, and testing for its accuracy.

The proposed Deuce framework is illustrated in Figure 1. Overall, the components of Deuce serve the same goal—to produce a seed set with high dual-diversity and informativeness.

In CSAL, data selection starts with only an unlabeled corpus. Deuce leverages PLM embeddings, which guide the selection process towards more diverse and informative samples.

Specifically, the embedding module implements a prompt-based, verbalizer-free approach (Jiang et al., 2022). This requires only a single inference pass per document.

However, raw embeddings suffer from template bias and length bias (Miao et al., 2023). Deuce further applies template denoising (Jiang et al., 2022) to obtain the denoised embeddings $z~$⁠.

This module aims to produce uncertainty-aware labels. With class information, Deuce gains prior knowledge about potential data distributions. With uncertainty information, Deuce is informed of potential labeling gain.

In CSAL, uncertainty represents the difficulty of an instance. Deuce adapts entropy, a common measure of uncertainty (§2.1.1).

Graphs serve as a powerful tool for data selection by explicitly modeling data interrelationship. This enables the propagation of valuable information (e.g., uncertainty) and the selection of more diverse samples. To integrate textual and class diversity, Deuce leverages manifold learning techniques (McInnes et al., 2020) on k-Nearest-Neighbor (kNN) graphs of both spaces.²

The use of kNN arises from the neighborhood perspective of diversity. Deuce aims to avoid selecting neighboring instances. In a kNN graph, an instance x_i is connected with its k nearest neighbors {x_ij} under some distance function Δ(·,·). Formally, the two metric spaces of kNN are defined as follows.

• The textual space $(X,Δz~)$ is defined by textual embeddings under cosine distance, $Δz~(xi,xj)=1πarccosz~xi⊤z~xj⊤$⁠;

• The label space $(X,Δŷ)$ is defined by label vectors under ℓ₁ distance, $Δŷ(xi,xj)=y^i−y^j1$⁠.

The kNN graph from each space is denoted by $Gz~$ and $Gŷ$⁠, respectively.

To unify textual and class diversity, Deuce merges the two kNN graphs into one for graph-based sampling. However, across two distinct spaces, it is necessary to first normalize the distances (McInnes et al., 2020).

To identify representative instances, Deuce performs graph clustering. This requires symmetric kNN graphs.

It is now appropriate to merge the two kNN graphs. This unifies textual and class diversity in one graph.

As merged, the DNG is an undirected graph $Gdual=(V,E,wdual)$⁠. The edges $E$ are the union of edges in $G~z~,sym$ and $G~ŷ,sym$⁠. Moreover, $E$ is divided into two types:

• $E1$ represents edges which only appear in either kNN graph, called single-neighbor edges;

• $E2$ represents edges which appear in both kNN graphs, called dual-neighbor edges. They connect neighboring documents which are similar in both textual semantics and class predictions.

Deuce adopts a hybrid acquisition strategy. Overall, the goal is to produce a diverse and informative seed set. To achieve this, the acquisition module performs graph clustering, propagation, and traversal on DNG.

A group of similar documents with high predictive uncertainty indicates an area where the model’s knowledge is lacking. By labeling one of the documents, the model predictions can be improved for similar ones in the area. Therefore, it is valuable to identify and prioritize such representatively uncertain (RU) groups for CSAL.

Clustering has been a common technique to group similar instances (§2.1.2). However, traditional clustering methods (e.g., k-means) are ill-suited, as the number of RU groups is unknown. Moreover, they force every instance into a cluster, while some instances may not belong to any RU group. Instead, Deuce adopts density-based clustering, which identifies RU groups with a sufficient density (≥ k_r similar documents).

Specifically, Deuce applies hdbscan* (Campello et al., 2013, 2015) on the DNG, with minimum cluster size k_r. A document x_i is either (a) clustered in an RU group c_l with membership p_i, or (b) excluded as a non-RU outlier.

The whole process is described in Algorithm 1.

Deuce is evaluated on six text classification datasets: IMDb (Maas et al., 2011), Yelp_full (Meng et al., 2019), AG’s News (Zhang et al., 2015), Yahoo! Answers (Zhang et al., 2015), DBpedia (Lehmann et al., 2015), and TREC (Li and Roth, 2002). Dataset statistics are shown in Table 1. All the datasets used in the experiments are publicly accessible. The original labels are removed to create a cold-start scenario.

To evaluate the performance of the acquired seed set $Xs$⁠, it is labeled and used for fine-tuning the PLM. The original labels of the seed set are revealed. The accuracy of the fine-tuned PLM on the test set is then reported. To be consistent with previous methods (Yu et al., 2023), the experiments adopt RoBERTa-base (Liu et al., 2019) as the backbone PLM.

The fine-tuning setup and hyperparameters are the same as Patron’s (Yu et al., 2023). Notably, the experiment code transplants the original implementation of graph normalization (McInnes et al., 2018) to GPU for acceleration. For Deuce, k = 500, k_r = 3, and γ = 1.0 (since $w~sym≤1.0$⁠) are taken. All experiments are run on a machine with a single nvidia A800 GPU with 80 GB of VRAM.

The following CSAL baseline methods are considered:

• Random sampling selects uniformly.

• Entropy-based uncertainty sampling (revisited by Schröder et al., 2022) selects data with the highest predictive entropy.

• Coreset selection (Sener and Savarese, 2018) iteratively selects data whose minimum distance to the selected data is maximized.

• alps (Yuan et al., 2020a) computes surprisal embeddings from BERT loss as uncertainty. They are then clustered with k-means. Data closest to each centroid are selected.

• few-selector (Chang et al., 2021) clusters the text embeddings with k-means.

• TypiClust (Hacohen et al., 2022) clusters the text embeddings with k-means, and selects data with the highest typicality, i.e., kNN density, from each cluster.

• Patron (Yu et al., 2023) clusters the text embeddings with k-means, and selects from each cluster data with the highest propagated uncertainty. It then iteratively updates the set to refine inter-sample distances.

• vote-k (Su et al., 2023) iteratively assigns a high score if a data is far from selected data.

Comparisons of the CSAL baselines and Deuce are presented in Table 2.

The main quantitative results of PLM fine-tuning performance with Deuce and baseline CSAL methods are shown in Table 3. Results for baselines other than vote-k are from Yu et al. (2023). To report the standard deviation, each setup is repeated with 10 different random seeds. Figure 2 demonstrates a qualitative visualization of the b = 128 seed set from IMDb dataset, acquired by the latest baseline method vote-k and the proposed Deuce. The t-SNE (van der Maaten and Hinton, 2008) method is used for visualization.

From results in Table 3, it can be seen that Deuce consistently outperforms other baselines, achieving up to a 2.5% gain on balanced datasets and up to 6.2% on the imbalanced dataset, TREC. Deuce mainly benefits from that it enhances the class diversity as well as textual diversity. This can be concluded from the larger improvements on TREC. In over half of the setups, Deuce also achieves the lowest standard deviation. In addition, Deuce improves most when b is small. This aligns with the fundamental goal of AL, which is to maximize performance gains with minimal labeled data. Furthermore, from the visualization in Figure 2, it can be seen that Deuce’s enhancement of dual-diversity leads to a broader and more balanced coverage of both input space and label space. As Deuce adopts a highest-uncertainty strategy, such coverage also exhibits high predictive uncertainty, thus including more “hard examples” which are valuable for annotation.

To verify the enhancement of class diversity, the class imbalance value (Yu et al., 2023) under b = 128 is reported in Table 4.

From Table 4, it can be seen that Deuce achieves the lowest average IMB value. This indicates that Deuce enhances class diversity properly. In contrast, an IMB of $∞$ emerges in the pure uncertainty-based (Entropy) and textual-diversity-based (Coreset) method. This indicates the missed cluster effect happens in their acquisition.

To measure the textual diversity of seed sets, the textual-diversity value (Ein-Dor et al., 2020; Yu et al., 2023) under b = 128 is reported in Table 5.

Table 5 shows that Deuce also achieves the highest average textual-diversity value. This indicates that Deuce also enhances textual diversity properly. The improvement of textual-diversity value is not significant, compared to IMB value’s (Table 4). This signals that Deuce enhances more of class diversity than textual diversity, compared to other baselines. Such difference can be explained by the highest-uncertainty-candidate strategy, which acquires more information from the label space.

To analyze the quality of Deuce’s prompt-based, unsupervised text embeddings $z~xi$ (§3.2.1), they are compared with the supervised Sentence Transformer embeddings (Sentence Transformers, 2024) used in vote-k (Su et al., 2023). The correlations are computed across all the possible $N2$ pairs of their cosine similarity.³ Results on three datasets are reported in Table 6.

From Table 6, a weak positive correlation is observed. Moreover, template denoising produces better embeddings, as it removes the biases from raw embeddings. Overall, the quality of textual embeddings is acceptable and adequate for cold-start acquisition.

To analyze the quality of embedding-based class prediction $y^i$ (§3.2.2), they are compared with gold labels. As uncertainty indicates unstable predictions, labels are arranged from the most confident (lowest u_i) to the least. Results are demonstrated in Figure 3.

From Figure 3, a high accuracy of class predictions is consistently observed with high confidence and with denoised embeddings, and vice versa. This demonstrates the good quality of e.d.f. predictions and the derived uncertainty metric.

The landscape of NLP is rapidly evolving with generative large language models (LLMs). This section evaluates two potential LLM-based alternatives to Deuce: serialization for acquisition and zero-shot Chain-of-Thought prompting. The following experiments are conducted with Llama 2 7B (Touvron et al., 2023).

Due to resource constraints, LoRA (Hu et al., 2022) is used for fine-tuning, with r = α = 64. Results are reported in Table 7. Despite utilizing a mid-sized PLM, Deuce outperforms serialization with LLM in most datasets. The decision process of LLM is also black-box. In contrast, Deuce adopts graphs to explicitly capture the interplay of information, offering better interpretability.

Zero-shot Chain-of-Thought (CoT) prompting (Kojima et al., 2022) with LLMs has emerged as a promising method in cold-start scenarios. This paper tests zero-shot CoT without and with explicit choices in prompts. The temperature of generation is set to 0, and a maximum of 256 tokens are generated. Results are shown in Table 8. From the results, fine-tuning PLM with Deuce still outperforms 0-shot LLM predictions. In class-imbalanced and difficult datasets, performance gaps are greater. Lemon-picking shows that the LLM failed to output a final answer within 256 tokens for many test instances.

In addition, the average total GPU and CPU energy consumption and time usage are measured using Alizadeh and Castor’s (2024) method. Results are reported in Table 9. There is a 7.82× difference in energy consumption and 6.26× in time consumption. While increasing the number of output tokens might improve, the added resource consumption cannot be neglected. Deuce provides an efficient solution for low-resource scenarios.

Real-world annotations often involve noise. Northcutt et al. (2021) estimated an average of 2.6% labeling errors across 3 commonly used NLP datasets. To evaluate Deuce under labeling noise, experiments with artificial errors are conducted. As the gold labels may already contain around 3% errors, 7% of seed labels are randomly replaced by wrong labels. The final sets are expected to exhibit an error level of 4–10%. Results are reported in Table 10.

From the results, a decrease in accuracy and an increase in standard deviation occur as expected. However, Deuce still outperforms 0-shot CoT (Table 8) in nearly all setups, despite the added noise. This shows the robustness of Deuce for fine-tuning to labeling noise.

For real-world cold-start tasks, the knowledge about classes might not be well exploited by the PLM. In the worst case, the PLM can fail to generate meaningful class predictions. To simulate this scenario, ablation experiments with random class predictions are conducted. In this setup, the predictive embeddings $zŷ∣xi$ are replaced with random vectors. This ablates class predictions. Results are reported in Table 11.

As class and uncertainty information are disarranged, Deuce degenerates to single textual diversity and performance degradation occurs as expected. Nonetheless, Deuce still outperforms Coreset selection (Sener and Savarese, 2018), a CSAL baseline which also purely utilizes textual diversity. This demonstrates Deuce’s effectiveness in real-world cold-start scenarios.

Deuce has the potential to generalize on other NLP tasks. To demonstrate this, Deuce is tested on GSM8K (Cobbe et al., 2021), a dataset of math word problems. However, directly adapting RoBERTa to solving math problems is difficult due to its masked modeling nature. Instead, Deuce is applied with RoBERTa to produce a seed set.⁴ Then, the seeds are taken as examples for few-shot Chain-of-Thought prompting (Wei et al., 2022) with Llama 2 7B. From the results, as reported in Table 12, Deuce is still effective in few-shot math problem solving, compared to random sampling.

This paper presents Deuce, a dual-diversity enhancing and uncertainty-aware CSAL framework via a prompt-based and graph-based approach. Different from previous works, it emphasizes dual-diversity (i.e., textual diversity and class diversity) to ensure a balanced acquisition. This is achieved by the novel construction of Dual-Neighbor Graph (DNG) and Farthest Point Sampling (FPS). DNG leverages the kNN graph structure of textual space and label space from a PLM. In addition, Deuce prioritizes hard representative examples, so as to ensure an informative acquisition. This leverages density-based clustering and uncertainty propagation on the DNG. Experiments show the effectiveness of Deuce’s dual-diversity enhancement and uncertainty-aware mechanism. It offers an efficient solution for low-resource data acquisition. Overall, Deuce’s hybrid strategy strikes an important balance between exploration and exploitation in CSAL.

Deuce leverages a discriminative PLM. However, state-of-the-art PLMs are primarily generative. Generative embedding models (e.g., Jiang et al., 2023) or adaptations (Yang et al., 2019; Gong et al., 2019; Zhang et al., 2022a) can be investigated and combined with Deuce. For such approaches, their quality and efficiency should be carefully minded.

In Deuce, the only source of external knowledge is the language model. Incorporation of more domain knowledge, if possible, can improve the performance in the cold-start stage. As Deuce adopts a prompt-based and graph-based acquisition, prompt engineering and knowledge graphs (Pan et al., 2024) can be investigated.

We extend our gratitude to our action editor, Sebastian Padó, and the anonymous reviewers for their constructive comments. We also thank Tianjun Li and Jiangfeng Liu for their helpful feedback on the initial drafts.

This work was funded in part by the National Natural Science Foundation of China grant under number 62222603, in part by the STI2030-Major Projects grant from the Ministry of Science and Technology of the People’s Republic of China under number 2021ZD0200700, in part by the Key-Area Research and Development Program of Guangdong Province under number 2023B0303030001, in part by the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (2019ZT08X214), and in part by the Science and Technology Program of Guangzhou under number 2024A04J6310.