Leveraging Continuous Prompt for Few-Shot Named Entity Recognition in Electric Power Domain with Meta-Learning

In recent years, China's electric power industry has developed rapidly and the scale of the industry has grown significantly. Under the influence of 5G, Internet of Things, and other high-tech, China's power industry has entered a new period of transformation and upgrading. The deep organic combination of cutting-edge artificial intelligence technology and traditional technology in the electric power industry has achieved good benefits. In the process of accelerating the production efficiency and economic transformation of the electric power industry, the knowledge graph has become an important way of electric power intelligence. In the whole construction process, named entity recognition (NER) of electric power text data is a fundamental part, and its purpose is to identify words or phrases in the text as different categories of entity labels [1], which provides the basis for the subsequent steps.

Pre-trained language models (PLM) [2] have been widely used for NER tasks. Existing PLM-based studies mainly consider NER as a sequentially labeled task with a label-specific CRF output layer added on top of the PLM representation. Although these approaches have achieved good results on open domain datasets, they still face the following two challenges in the Chinese electric power domain: (1) Lack of training instances. Due to the high cost of labeling, for an entity type, models often rely on less than one hundred instances for training. This few-shot scenario is a serious challenge to the generalization ability of the model. (2) Unseen new entity types. New entity types are very common due to the frequent expansion of business scenarios. Since the CRF output layers of existing methods are fixed, they have to retrain the models to predict new entity types. This not only does not meet the efficiency requirements of the electric power industry but also does not guarantee good recognition.

To address the above challenges, this paper proposes a Chinese NER model (Meta Continuous Prompt NER model, MCP-NER) based on meta-learning and continuous prompt-tuning. The model is based on a generative PLM in order to directly predict new types of electric power entities without modifying the model structure. To make up for the lack of training instances, MCP-NER applies prompt-tuning to mine the potential prior knowledge in PLM. Different from previous works using the manual prompt template[9], MCP-NER sets continuous prompt vectors to direct the attention flow of the transformer to avoid optimizing the entire parameters of the PLM, thus greatly improving the training efficiency. To further improve the generalization capability on unseen types, MCP-NER is trained by a meta-learning algorithm. It enforces the model to capture the common feature of different entity types by correcting the gradient between the support set and query set. We constructed a Chinese electric power NER dataset using real national grid data and conducted a comprehensive experiment using it. The experimental results show that our MCP-NER method outperforms all comparative methods and achieves state-of-the-art performance in few-shot NER scenarios.

The rest of this paper is organized as follows. In Section 2, we will introduce some related work of few-shot NER. In Section 3, we will formulate the few-shot NER problem. In Section 4, we will detail our proposed MCP-NER and its motivation. In Section 5, we will introduce our experimental setup and results. In Section 6, we will present our conclusions and look ahead to future work.

There are two important lines of research for few-shot NER: PLM-based approaches and meta-learning-based approaches.

PLM has recently had a significant impact on NER [2, 3], where Transformer-based models [2] are used as the core network structure, gaining widespread attention. The current mainstream methods [4, 5, 6] all regard NER as a sequence labeling problem. Yan et al. [5] proposed the Unified-NER model, which formulates the NER subtask as an entity-span generation task, leveraging hand-crafted one-to-one mappings from tokens in the vocabulary to entity types; however, it fails to generate complex entity types that correspond to, such as business requirements and fault exceptions in the electric power domain.

Prompt-based optimization methods [3, 4] have received a great deal of attention since the advent of GPT-3. GPT-3 shows that large-scale language models can achieve superior performance in few-shot scenarios through fine-tuning and context learning. Schick et al. [7] argue that small-scale language models can also achieve good performance using prompt optimization. While most research has been done on text classification tasks, some work extends the impact of prompt optimization to other tasks, such as relation extraction. In addition to various downstream tasks, cues are used to probe knowledge from PLMs and are thus a promising learning paradigm [15, 16, 17, 18, 19, 33, 34].

Recently, Cui et al. [9] proposed a prompt-based Template-BART model for few-shot NER, which enumerates all possible spans in a sentence and populates them into hand-crafted templates. Then the output of the model is to classify each candidate entity span according to the corresponding template score. Unlike their approach, the MCP-NER model proposed in this paper does not rely on template engineering but uses learnable prompt embeddings to guide attention to guide knowledge in PLM more efficiently. In addition, the MCP-NER model only updates lightweight prompt vectors during training, which greatly improves the efficiency of PLM-based methods. Furthermore, some works in related fields have used soft prompts [28, 29, 30, 31, 32] achieve the good performance. Inspired by them, we apply this technique to the NER task.

Meta-learning has become a popular method in research on few-shot learning. This type of method [11, 12] utilizes the nearest neighbor criterion to assign entity types. Typically, this depends on the similarity patterns of entities between the source and target domains without updating the network parameters for the NER task. However, this also makes them unable to improve neural representations of cross-domain instances. Other studies have utilized transfer learning methods [13, 14] for knowledge transfer across languages or domains to enhance few-shot learning. MetaNER [20] incorporates meta-learning and adversarial training strategies to encourage robust, general, and transferable representations for sequence labeling. de et al. [21] propose a task generation scheme for converting classical NER datasets into the few-shot setting, for both training and evaluation. Ma et al. [22] proposes MAML-enhanced prototypical networks to find a good embedding space that can better distinguish text span representations from different entity classes. Li et al. [23] propose FewNER separates the entire network into a task-independent part and a task-specific part, in order to cope with them separately. Unlike them, our approach builds on prompt-tuning, where the operations in meta-learning are applied to prompt vectors, which are more lightweight and flexible.

Given an input text sequence X = (x⁰, x¹, …, x^n-1}, the goal of NER is to output a set E = {e¹, e², …, e^m}, where n is the total number of words, m is the number of recognized entities, xⁱ represents the i-th word in the text, and eⁱ = (uⁱ_stalt,uⁱ_end,c’) represents the recognized i-th entity. Here, uⁱ_start ∈ [1,n] and uⁱ_end ∈ [uⁱ_start,n] represent the start index and end index referred to eⁱ in X respectively, cⁱ ∈ C represent the type label of eⁱ. C is the set of all type labels. The training dataset for NER often consists of pair-wise data D_train = {(X¹, E¹), …, (X¹, E¹)}, where l is the number of training instances. Traditional NER systems are trained in the standard supervised learning paradigms, which usually require a large number of pairwise examples, i.e., l is large. In real-world applications, the more favorable scenarios are that only a small number of labeled examples are given for each entity type (l is small) because expanding labeled data increases annotation cost and decreases customer engagement. This yields a challenging task few-shot NER.

In order to enable the model to deal with unseen entity types during testing without changing its own structure. For sequence X = (x⁰, x¹, …, x^n-1}, we first convert the original target E into sequence T = {t⁰, t¹, …, t^1-1}, where r is the total number of indexes, each index tⁱ corresponds to a unique token ĩⁱ

Here C denotes the set of all entity types. When 0 < tⁱ < n, $t~i$ is the tⁱth word in the input text X, which is used to form the identified entity mention; when n ≤ tⁱ < n + |C|, $t~i$ is the tⁱ – nth entity type label C^ti, which is the type corresponding to the identified entity mention; suppose X = {{已,检,查,电,表,无,故,障,何, 时,可,以,复,电} (The energy meter has been checked. When can the electric power supply be restored) which contains two entities: “电表 (Energy meter)” and “复电 (Restoring electric power supply)”, then the target index sequence is T = (3,4,14,12,13,17}. Among them, 3 and 4 represent the indexes of the two words “电” (electric power) and “表 (meter) in X respectively, 14 represents the sum of the index 0 in C and the text length n = 14 of the entity type [machine equipment] of “电表 (Energy meter)”; 12 and 13 respectively Indicate the index of “复” (restore) and “电” (electric power) in X, and 17 represents the sum of the index 3 in C and the text length n = 14 of the type “复电 (Restoring electric power supply)” [Business requirements].

For any input X, its corresponding generation probability of T can be modeled as:

where P(tⁱ | X, t⁰, t¹,…,t^{i − 1}) represents the probability of generating tⁱ by combining X and the generated t⁰, t¹, …, t^{i − 1}.

The structure of the MCP-NER model proposed in this paper is shown in Figure 1. It uses the PLM model BART to model P(tⁱ | X, t⁰, t¹,…,t^{i − 1}), which consists of an Encoder and a Decoder, and autoregressively generates an index sequence T, as shown below:

Among them, $Hseq∊Rnxd$ and $Htype∊R|C|×d$ represent the semantic matrix of X and C respectively, and $hi∊Rd$ is the hidden state of the moment ¡ in the decoding process, which is used to calculate the probability distribution Pⁱ of the index tⁱ:

Among them, $PSeqi∊Rn$ and $Ptypei∊R|C|$ represent the probability distribution vectors of entity reference and entity type generated at moment i, respectively. [$Pseqi$ · $Ptypei$] represents the concatenation of vectors $Pseqi$ and $Ptypei$ to obtain the overall probability distribution $Pi∊Rn+|C|$ as output.

Since there is no output layer with a fixed number of categories, for the new entity type c’ in the prediction process, only need to add its label to the set C, MCP-NER can calculate the new probability distribution Pⁱ, and avoid the retraining caused by adjusting the output layer.

The PLM model contains prior knowledge learned from massive corpora, which can effectively make up for the lack of training data. However, the huge number of parameters in PLM makes its training time too long and the optimization efficiency is low. In order to use PLM to alleviate the few-shot challenge in Chinese electric power NER while ensuring the training efficiency of the model, MCP-NER proposes to incorporate vectorized prompt into the self-attention layer of the BART model to guide the distribution of attention. During model optimization, only these prompt vectors are updated, not the parameters of the PLM.

Specifically, MCP-NER provides two sets of trainable parameters φ =(φ¹, φ², …, φ^ν} for the encoder and the decoder respectively, where $φi=[φKi;φVi]∊R2×Z×d$⁠, and N represents the number of layers of the encoder or decoder, and Z is an adjustable hyperparameter representing the number of prompt vectors in each layer (set to 10 in the experiment). In fact, some related works have proposed other ways to apply continuous prompt, such as using prompt as a prefix or suffix for the input word vector of the encoder. Here we only catenate the soft embeddings at the end of K and V because we want the continuous prompt to be directly involved in the attention stream of the encoder and decoder to achieve a more fine-grained prompt.

As shown in Figure 2, the input vector sequence Xⁱ ∈ R^d of the ith layer is first projected into three dimensions: query, key, and value.

W_Q, W_K, W_V ∈ R^{d × d} is the original parameter matrix in BART, which is fixed during training. Subsequently, MCP-NER modifies the attention mechanism and introduces prompt vectors as shown below.

Among them, $φKi∊RZ×d$ and $φVi∊RZ×d$ represents the parameter vector added to the key and value respectively. These vectors serve as prompt, concatenated with Kⁱ and Vⁱ to participate in the attention mechanism. MCP-NER aggregates and computes attention scores to guide the final attention flow.

In the experiment, the parameter size of the prompt vector is much smaller than that of the whole BART, so the model training time can be greatly shortened. At the same time, since these prompts are deeply involved in the attention flow, the prior knowledge in the BART parameters can be fully utilized to ensure the model's few-shot learning ability.

In order to further improve the few-shot learning ability of the model and enable it to quickly adapt to new entity types that have not been seen during training, this paper proposes a Meta-Learning strategy to train MCP-NER.

First, the few-shot NER training set is divided into several tasks, where each task G consists of several training samples {(X₁, E₁), …, (X_|G|, E_|G|)}. The samples in the same task cover N entity classes, and each entity class has K samples. During training, these samples are divided into two sets, called support set S and query set Q respectively. Subsequently, for each G, MCP-NER updates its prompt vector parameter φ through the following process:

1. Input each sample in the support set S into the model M_φ, calculate the support loss L_S(M_φ(X),T), and then get the gradient of the prompt parameter ∇_φL_s;

2. Use the gradient descent method to get the temporary prompt parameter $θ=φ-α∇φLS$⁠, where α is the learning rate;

3. Input each sample in the query set Q into the model, and calculate the query loss L_Q(M_θ(X),T);

4. Combine the loss of the support set and the query set to get the overall loss $LT=σLS+(1-σ)LQ$⁠, where σ is the weight hyperparameter;

5. Use the SGD optimization algorithm and the learning rate β to optimize the original prompt parameters φ to get the final prompt parameters $φ*=φ-β∇φLT$⁠.

Here, M_φ(X) represents the predicted output of the MCP-NER model for the input sequence X, T is the corresponding target output, and all loss functions L use cross-entropy loss. In order for the model to have the ability to recognize new types of entities, the training process ensures that the entity types in the support set S and query set Q are disjoint to simulate this situation. According to this split, the model undergoes a two-stage gradient update during the training process for each task. The first stage is to make the model fit the entity type in S, and get the optimal parameters θ on S. This parameter is used as a possible optimization direction, but the original parameters are not updated at this time. In the second stage, Q is used to rectify θ, by optimizing the joint loss L_T to make the model focus on common features in Q and S to adapt to new entity types not seen in Q. Finally get updated φ*, which is better at capturing common features in different entity types, so that the sample size can be smaller or even missing.

The hardware environment of the experiment in this paper: the CPU adopts Intel® Core 7700, the memory is 32GB, and the GPU adopts Nvidia RTX-2080ti. Software environment: ubuntu 18.04, Python 3.6.8, deep learning framework adopts Pytorch 1.4.0.

In this article, we collected data of 18,000 work orders from the customer service of State Grid Jiangsu Company in China between May and June 2021. We hired outsourcers to label the “User Acceptance Content” part for all of the data according to the pre-defined NER specification, and finally constructed the named entity recognition dataset ENER. In this dataset, we define the following 13 types of entities:

The training set, validation set and test set contain 1,358, 1,059 and 2,032 instances, respectively. For each entity type, the number of the training instances is less than 100.

In addition, we use the CoNLL 2003 [27] as an open-domain dataset to evaluate the performance of our model on conventional benchmarks.

In this paper, the accuracy rate (P), recall rate (R) and F₁ score (F₁) are used as the evaluation indicators of model performance to evaluate the entity recognition results. The calculation methods are:

Among them, T_TP represents the number of entities correctly recognized by the model; F_TP represents the number of entities recognized but actually wrong; F_FN represents the number of entities that are actually correct but not recognized by the model.

In this paper, several tradtional baseline models commonly used in NER are used as the comparison methods for MCP-NER: BiGRU, BiLSTM-CNN, and BiLSTM-CRF. These methods use a recurrent neural network (RNN) as the encoder and a CRF or convolutional neural network (CNN) as the output classifier. In addition, this paper compares with PLM-based methods that have performed well in recent years: Sequence Labeling-BERT (SL-BERT) [2], Sequence Labeling-BART (SL-BART) [24]. We also compare our method with few-shot learning methods: FewNER [25], Model Fusion [26]. They utilize a linear layer to classify the hidden vectors at each input location, resulting in NER labels. Finally, this paper compares with the latest prompt template optimization method Template-BART [9]. This method applies templated-based prompt to utilize the PLM.

In order to test the contribution of continuous prompt optimization and meta-learning training strategy in MCP-NER to the overall effect, this paper designs the following two groups of ablation experiments: (1) MCP-NER (w/o prompt): The prompt vector of the model species is deleted, and only the Fine-tune BART generation results, at which point all parameters in BART are no longer fixed. (2) MCP-NER (input prefix prompt): The prompt vector is directly prefixed to the encoder's input, not at the end of K and V. (3) MCP-NER (w/o ML): Use the Mini-Batch strategy commonly used in machine learning instead of the meta-learning strategy to train the model.

First, all methods are trained using the full training set of ENER. The experimental results are shown in Table 2. Among them, SL-BERT and SL-BART, as strong baselines, show good results. Nonetheless, MCP-NER still achieves the best results, demonstrating the effectiveness of this paper in handling few-shot scenarios. Furthermore, although both MCP-NER and Template-BART are based on BART, the former achieves (2.42% vs. 0.82%) and (3.47% vs. 2.12%) higher F1 scores than these two methods, respectively, which proves that continuous prompt are more effective than prompt templates. Correspondingly, ablation experiments also demonstrate the contribution of prompt optimization and meta-learning strategies. In contrast, meta-learning improves the ability of few-shot learning even more.

Figure 3 lists the training time of MCP-NER and several typical comparison methods when using the full training set of ENER. Among them, the abscissa is the comparison method, and the ordinate is the training time (minutes). To ensure fairness, all methods are trained until the model converges. PLM-based methods require a long training time due to their huge amount of parameters. Among them, Template-BART takes the longest because it also needs to enumerate entity spans and entity types. On the contrary, the MCP-NER in this paper only needs to optimize a small part of the prompt parameters, thus greatly reducing the time overhead of training, even close to the BiLSTM-CRF model that does not use a pre-trained model.

In order to further explore the influence of the number of samples on MCP-NER, for each entity type, {10, 20, 50} training instances were randomly sampled as D₁₀, D₂₀ and D₅₀ from the original training set of ENER respectively. Following the N-way-K-shot settings, we also change the ways (i.e., the number of the entity types) for different settings of the shots. Train on these three subsets and test the effect on the original test set of ENER. The experimental results are shown in Table 3. The fewer training samples, the more obvious the improvement of MCP-NER compared to other methods. This is because prompt optimization aligns PLM pre-training with the goal of NER, which can guide rich prior knowledge to cope with few-shots. In addition, when the training samples are small enough, the improvement brought by meta-learning is very obvious, which fully proves its effectiveness.

In order to explore the transferability of MCP-NER's recognition ability to unseen entities in the training set, this paper samples from the original training set of ENER and constructs a source domain training set D_S, and three target domain test sets D_T¹, D_T², and D_T³. Among them, D_S contains all the training samples of the following 10 entity types: Financial Bills, Electronic Channels, User Information, Document Regulations, Marketing Activities, Identity, Company, Illegal Acts, Professional Vocabulary, Electricity price; while D_T¹, D_T², and D_T³ contain test data of types Machine Equipment, Business Requirements and Fault Exceptions, respectively, which are used to simulate the data of new types of entities. In this experiment, all models are trained on D_S, and then tested on the test sets of D_L¹, D_L², and D_L³ respectively.

The experimental results are shown in Table 4. The MCP-NER model has the best performance in recognizing entities of unseen types during training, validating the effectiveness of the generative framework and meta-learning approach. Among them, the effect is the best on the Business Requirement type, which may be because the Business Requirement is closer to the entity type in the source domain, and the domain of Machine Equipment and Fault Exception are more different, so the improvement is relatively small. In addition, regardless of the entity type, MCP-NER(-prompt) and MCP-NER(-weight) always have a left-right effect gap compared with the overall model, which proves that continuous prompt optimization and meta-learning strategy are also effective in cross-type transfer learning.

In this paper, a method based on meta-learning and continuous prompt optimization is proposed for the few-shot NER task in the Chinese electric power domain. Different from traditional approaches based on PLM models, this method uses trainable parameter vectors as prompts instead of fine-tuning the entire model.

The entire model is based on pre-trained BART and uses autoregressive decoding to generate entities and corresponding types. Only parameter vectors are added as prompts between the encoder and decoder to guide the optimization of the entire model. Furthermore, to cope with new types of entities that have not been seen before, the model is trained with a meta-learning-based strategy.

Experiments show that, our method achieves the best results in few-shot scenarios. Moreover, as the number of samples decreases, the improvement of the proposed method is more significant than that of the comparison method. In future work, we consider combining in-context learning with prompt-tuning, using the training data as a direct prompt.

This work is supported by the Science and Technology Project (No.J2021151) of State Grid Jiangsu Co., Ltd.

Yang Yu ([email protected], 0000-0003-4180-7970) was responsible for researching and proposing the overall methodology, conducting the main experiments, and writing the article.

Wei He ([email protected], 0000-0002-7993-4441) assisted in some of the experiments and was responsible for writing and revising the article.

Yu-meng Kang ([email protected], 0000-0003-3538-0085) assisted in some of the experiments, and was responsible for statistical analysis and article writing.

You-lang Ji ([email protected], 0000-0002-2736-3317) assisted the experiments, and was responsible for writing and revising the article.