Planning with Learned Entity Prompts for Abstractive Summarization

Jones (1993) described text summarization—the task of generating accurate and concise summaries from source document(s)—as a three-step process: (i) Building the source representation from the source document(s), (ii) Learning a summary representation from the source representation, and (iii) Synthesizing the output summary text. Common to most traditional methods, an input representation was learned by semantically analyzing the source text, the summary representation was then learned by modifying and refining the input representation, and finally the summary was generated grounded to the intermediate summary representation Luhn, 1958; McKeown and Radev, 1995; Barzilay and Elhadad, 1997; Mihalcea and Tarau, 2004.

State-of-the-art neural summarizers are powerful representation learners and conditional language models, thanks to sequence-to-sequence architectures (seq2seq) with attention and copy mechanisms Hochreiter and Schmidhuber, 1997; Bahdanau et al., 2015; See et al., 2017, Transformer architectures with multi-headed self-attention (Vaswani et al., 2017), and large pretrained conditional language models (Dong et al., 2019; Song et al., 2019; Lewis et al., 2020; Rothe et al., 2020; Raffel et al., 2019; Zhang et al., 2020). However, the grounding of summary generation that was inherent to most traditional methods is yet to be achieved in neural summarization. The attention mechanism (Bahdanau et al., 2015), especially in pretrained encoder-decoder models (Lewis et al., 2020; Raffel et al., 2019; Zhang et al., 2020), plays a key role in aligning summary content to the input, yet undesired hallucinations are common in generated summaries (Maynez et al., 2020; Kryscinski et al., 2020; Gabriel et al., 2021).

In this paper, we investigate Entity Chains— ordered sequences of entities¹ in the summary— as an intermediate summary representation to better plan and ground the generation of abstractive summaries. During training, we construct an augmented target summary by extracting and prepending its corresponding entity chain (Figure 1, right). At test time, the model must generate both the entity chain followed by the summary. Concretely, we use Transformer-based encoder- decoder (Vaswani et al., 2017) models; a transformer encoder first encodes the input and a transformer decoder generates (i) an intermediate summary representation in the form of an entity chain; and (ii) the summary conditioned on the entity chain and the input. We evaluate our approach on four popular summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019), and BillSum (Kornilova and Eidelman, 2019), and show that the state-of-the-art Pegasus (Zhang et al., 2020) pretrained models finetuned with the planning objective clearly outperform regular finetuning in terms of entity specificity and planning in generated summaries on all datasets. We further demonstrate that this simple planning mechanism can be easily used for pretraining summarization models to do entity-level content planning and summary generation. Similar to Pegasus pretraining, we mask important sentences from an input document, extract an entity chain from the masked sentences, and generate these gap-sentences prepended with their entity chain from the rest of the document (Figure 1, left). We see further gains with pretraining achieving state of the art performance on XSum in terms of rouge. We further demonstrate how the entity-level content planning in summarization can be easily leveraged to mitigate hallucinations in abstractive summaries. In particular, we modify the predicted entity chain to only keep entities that are seen in the document and then generate the summary prompted with the modified entity chain, outperforming state- of-the-art approaches when evaluated automatically and by humans for faithfulness. Our main contributions are as follows:

We introduce a novel training objective to neural summarization models for content planning with entity chains, and study integrating it with supervised finetuning and self-supervised pretraining objectives without altering the models themselves. As the entity chains are extracted from the reference summaries during training, our models learn to ground the generation of summaries to the entity chains found in them. Hence, we refer to this objective by Frost for its ability to try to “FReeze entity-level infOrmation in abstractive SummarizaTion with planning.”

Frost provides a very effective knob for entity-level content modification in abstractive summaries. In this paper we empirically demonstrate how Frost is critical for faithfulness by enabling the drop-prompt mechanism where we drop out hallucinated entities from the predicted content plan and prompt the decoder with this modified plan to generate faithful summaries. We further qualitatively demonstrate that Frost enables generation of summaries (i) with topical diversity by choosing different sets of entities from the source to plan what we want to discuss in the summary, and (ii) with style diversity by reordering entities in the predicted plan to get an equivalent summary but with a different entity emphasis.

Our codes to process summarization datasets to do Frost-style content planning and generation, models and predictions are available at https://github.com/google-research/google-research/tree/master/frost.

Traditional methods argue on the granularity of linguistic, domain, and communicative information included in the source representation needed in order to plan and build better summary representations. Some argued to use a deep semantic analysis of the source text, such as Rhetorical Structure Theory (RST; Mann and Thompson, 1988) or MUC-style representations (McKeown and Radev, 1995) to interpret the source texts, while others used a shallow semantic analysis using only word frequency (Luhn, 1958) or lexical chains (Barzilay and Elhadad, 1997).

Recent encoder-decoder models for text generation (Bahdanau et al., 2015; Sutskever et al., 2014; Vaswani et al., 2017; Rothe et al., 2020; Lewis et al., 2020; Raffel et al., 2019; Zhang et al., 2020) tend to perform text generation in an end-to-end setting, which means that most approaches do not explicitly model content planning. Wiseman et al. (2018) and Hua and Wang (2020) start the generation process by building templates which are then used for realization. In data-to-text generation, Puduppully et al. (2019) generate a content plan highlighting which information should be mentioned in the table and in which order. In story generation, there has been some work on exploring events (Martin et al., 2018) or sequences of words (Yao et al., 2019) to plan ahead when creating a consistent story. We are not aware of any similar work on content planning for summarization using encoder-decoder models.

Pretrained transformer-based models have dramatically changed the text generation space, and summarization is no exception to this. Most models focus on task-agnostic pretraining using the left-to-right language modeling objective (Radford et al., 2018; Khandelwal et al., 2019; Dong et al., 2019) or reconstructing the corrupted input text using a sequence-to-sequence framework (Song et al., 2019; Lewis et al., 2020; Raffel et al., 2019). There have been few attempts towards task-specific pretraining for summarization to teach models to do better content selection. Zhang et al. (2020) proposed to select important sentences from an input document as a proxy for human-authored summary and then to generate them from the rest of the document. Narayan et al. (2020) proposed question generation pretraining to better align with summarization. To the best of our knowledge we are the first to propose a solution that incorporates content planning directly into pretraining.

There is a growing interest in enabling users to specify high-level characteristics such as length, keywords, and topic in order to generate summaries that better suit their needs. In most cases, these features are first manually provided or estimated using third-party content selectors, and then either (i) encoded along with the input (Fan et al., 2018; He et al., 2020; Dou et al., 2021) or (ii) used to filter beams for lexically constrained decoding (Mao et al., 2020), to control the summary generation. On the contrary, our approach is more generic as we do not rely on external systems or data to augment the input; users can prompt the decoder with a desired content plan in the form of an entity chain to control the summary.

We introduce a new training objective for encoder-decoder generative models to do content planning while summarizing.

Let d be an input document, we aim to teach our model to first generate a content plan c for the summary s as p(c|d), and then generate the summary s as p(s|c,d). We define the ordered chain of entities observed in the summary s as its content plan. Instead of modeling p(c|d) and p(s|c,d) separately, we take a simpler approach, we train an encoder-decoder model to encode the document d and generate the concatenated content plan and summary sequences c;s, essentially the decoder first predicts the entity chain c and then continues predicting the summary s using both c and d. We prefix c and s with special markers “[ENTITYCHAIN]” and “[SUMMARY]”, respectively, as shown in Figure 1. If s consists of multiple sentences, we use sentence markers “|||” to mark them in c. The model is trained with the standard maximum-likelihood objective generating the augmented target c;s.

We modified Pegasus (Zhang et al., 2020) to pretrain our models for entity-level content planning and summary generation.² In particular, we select a maximum of n most important sentences using self-rouge from an input document, the selected sentences work as a proxy for a human-authored abstractive summaries for the rest of the document. We construct a target by prepending the selected sentences dynamically with their entity chain. Our model is then trained to generate this target from the rest of the document.³

As entities in the summary contribute to the continuity of lexical meaning of the summary, we hypothesize that by learning to predict the entity chain c in advance, we enforce our model to learn entity-level lexical cohesion (Barzilay and Elhadad, 1997) and coherence (Halliday and Hasan, 1976; Azzam et al., 1999) in the summary. We hope that by doing so our model will be better at predicting pertinent entities (entity specificity) in their right order (entity planning) in generated summaries; the prediction of entities in the entity chain c (as p(c|d) in Frost) will be less susceptible to local correlations compared to when predicting them directly in the summary s (as p(s|d)). Furthermore, as c is predicted in advance and the generation of s is grounded to c,⁴ our model will be better equipped to predict correct events relating to different entities in c with full access to c and not just entities to the left, already decoded.

An advantage of training to generate the summary s following the generation of the plan c using the same decoder is that now during the inference time the decoder can be easily prompted with any desired content plan c′ to control the content in the output summary s′ with a probability of p(s′|c′,d). In Section 5, we prompt our decoder with modified content plans to mitigate hallucinations and to generate diverse summaries.

We experiment with both base and large transformer architectures (Vaswani et al., 2017). The base architecture has L = 12, H = 768, F = 3072, A = 12 (223M parameters) and the large architecture has L = 16, H = 1024, F = 4096, A = 16 (568M parameters), where L denotes the number of layers for encoder and decoder Transformer blocks, H for the hidden size, F for the feed-forward layer size, and A for the number of self-attention heads. All pretrainings are done with a batch size of 1024, whereas all finetuning experiments are done with a smaller batch size of 256. For optimization, we use Adafactor (Shazeer and Stern, 2018) with square root learning rate decay and dropout rate of 0.01 during pretraining and 0.0001 during finetuning. All finetuned models were decoded with a beam size of 8 and a length-penalty of 0.8.

Following Zhang et al. (2020), our model pretraining also relied on two large Web corpora which were processed to look like plain text: (i) C4 (Raffel et al., 2019) is composed of 350M Web pages that were obtained from Common Crawl, and (ii) HugeNews (Zhang et al., 2020) is composed of 1.5B news and news-like articles from 2013–2019. This dataset includes articles from multiple allowlisted sources including news publishers, high-school newspapers, and blogs.

We evaluate our models on four summarization datasets: CNN/DailyMail highlight generation (Hermann et al., 2015), XSum extreme summarization (Narayan et al., 2018), SAMSum dialogue summarization (Gliwa et al., 2019), and BillSum summarizing US Congressional bills (Kornilova and Eidelman, 2019). We use the publicly available versions through the TFDS Summarization Datasets.⁵ We use the original train/validation/test splits for them. For BillSum, where the validation split was not provided, we split 10% of the training set to serve as validation. Inputs and outputs were truncated to 512 and 128 for XSum and SAMSum, and 1024 and 256 for CNN/DailyMail and BillSum. Table 1 provides more insights into these datasets to better understand the effect of summary planning with entity chains.

We experimented with entity chains consisting of named entities, dates, and numbers. We annotate the whole document in the pretraining datasets to allow dynamic construction of summaries and their entity chains during pretraining. We only annotate the reference summaries for the finetuning summarization datasets. We use a BERT-based tagger trained on CoNLL-2003 data (Tjong Kim Sang and De Meulder, 2003) to identify named entities, and regular expressions to identify dates and numbers (Guu et al., 2020). See Table 1 for the number of named entities, dates and numbers found in different datasets.

Tables 6 and 7 in Appendix A present hyperparameters used for pretraining and finetuning Pegasus and Frost base and large-sized models. We used Cloud TPU v3 accelerators for training.

We evaluate our Frost models on rouge, entity specificity, entity planning, faithfulness using automatic and human evaluations, and overall quality by humans. Our models predict a summary plan in the form of an entity chain, followed by a summary. All evaluations are done on the summary, the predicted entity chains are stripped out.

We report rouge F1 scores (Lin and Hovy, 2003) to assess generated summaries.⁶

We evaluate the quality of content plans learned by our models by assessing the entities and the order in which they appear in the summary. We annotate both predicted and reference summaries with named entities, dates, and numbers, and report on rouge F1 scores for entity chains found in the predicted summaries against corresponding reference entity chains.

We compare entities in predicted and reference summaries, and report average entity F1-scores (EntF1) in predicted summaries. We lower-case and remove duplicate entities; we report on the exact match with reference entities.

For entity-level faithfulness, we report on EntPrec, measuring the precision of entities in predicted summaries against their input documents. EntF1, in comparison, evaluates predicted summaries for entity specificity against reference summaries and is not a measure for faithfulness. For summary-level faithfulness, we follow Maynez et al. (2020) and report on textual entailment (Pasunuru and Bansal, 2018; Falke et al., 2019; Kryscinski et al., 2020). In particular, we report the probability of a summary entailing (Entail.) its input document using an entailment classifier trained by fine-tuning an uncased BERT-Large pretrained model (Devlin et al., 2019) on the Multi-NLI dataset (Williams et al., 2018).

We further assess summary faithfulness by humans. Our annotators, proficient in English, were tasked to read the document carefully and then grade its summary on a scale of 1–5 (fully unfaithful, somewhat unfaithful, 50-50, somewhat faithful, and fully faithful); a summary is “fully faithful” if all of its content is fully supported or can be inferred from the document.

Finally, we assess the overall quality of summaries by humans. We ask our annotators to read the document carefully and then grade its summary on a scale of 1–5 (poor summary, better than poor, okay summary, better than okay, and great summary). To improve the annotator agreement, we clearly define 4 features that a “great summary” should have: Relevant (summary should have most important information from the document), Accurate (summary should be accurate with respect to the document or the expert knowledge),⁷Concise (summary should not have redundant or less-important content), and Fluent (summary should be grammatically correct and coherent). A great summary should pass on all 4 features, a better than okay should have 3 out of 4 features, and so on.

For both human assessments, we collected 3 ratings for each (document, summary) pair and report the average of all assigned labels (1–5) to a system. We conducted 3 pilot studies for each setup to train our annotators with examples to improve their understanding of the task. Additionally, extra measures were taken to improve agreements among our annotators. For example, for the faithfulness assessment, when one of somewhat unfaithful, 50-50, and somewhat faithful were selected, annotators were asked to also specify what was faithful or unfaithful in the summary. Similarly for the overall quality assessment, when one of better than poor, okay summary, and better than okay were selected, they were asked to list all features on which the candidate summary fails. Figures 9 and 10 in Appendix B show detailed instructions for human evaluations for faithfulness and overall quality of summaries, respectively.

Let the summary s consists of m sentences s₁…s_m and c_i be the entity chain for the sentence s_i, we consider generating the summary s in two ways. Sentence-level approach trains a model to generate s by consecutively generating the sentence-level content plan c_i followed by its summary sentence s_i with a probability p(c_is_i|c₁s₁…c_i−1s_i−1;d); d is the input document. Summary-level approach first generates a summary-level content plan c = c₁|||…|||c_m and then continue generating the summary s with a probability p(s|c;d); ||| are sentence markers. The summary-level planning is arguably better suited for summarization than the sentence-level planning. By planning the whole summary beforehand, the summary-level planner would be (i) less susceptible to local correlations than the sentence-level planning while generating the entities, and (ii) a better microplanner in deciding sentence boundaries and avoiding verbose summaries (Reiter and Dale, 1997). See examples for sentence-level and summary-level entity chains in Figure 2.

We finetuned our large models initialized with the Pegasus checkpoint on the CNN/DailyMail dataset for sentence-level vs summary-level entity chain ablations. We did not do this study on the XSum dataset as the XSum summaries are single sentences, which means that sentence-level and summary-level entity chains are the same. In contrast, the CNN/DailyMail summaries consists of multi-sentence highlights. Results are presented in Figure 3.

We found that the summary-level planning is not only superior to the sentence-level planning, it helps with generating better quality summaries in terms of summary-level rouge (RL-Sum), entity planning (R2-EPlan), and entity specificity (EntF1). For the rest of the pretraining or finetuning experiments, we focused on summary-level planning unless specified otherwise.

We pretrained three base-sized models from scratch: Pegasus-base pretrained with the original gap-sentence objective (Zhang et al., 2020) for 1.5m steps, Frost(F)-base pretrained with the gap- sentences prepended with their entity chain for 1.5m steps, and Frost(P+F)-base first pretrained with the Pegasus objective for 1m steps and then with the Frost objective for another 500k steps. Maximum input and output lengths were set to 512 and 256 sentencepieces during pretraining, respectively. We finetuned these three models on the XSum dataset. Results are shown in Figure 4.

First of all, our results confirm that the pretraining for planning is beneficial for summarization; both Frost(F) and Frost(P+F) learned better content plans (in terms of entity chain-level rouge and EntF1) than Pegasus without sacrificing the summary quality (in terms of summary-level rouge) significantly. Interestingly, the Frost(P+F) finetuned model outperformed Frost(F) on all measures confirming that the pretraining for planning and summarization is more effective when started with summarization pretrained models such as Pegasus than when trained jointly from scratch. Hence, for our future pretraining experiments we initialize our model with existing Pegasus checkpoint and continue pretraining for content planning and refer to them as Frost for simplicity.

Based on our findings, we pretrain a large Frost model for planning for summarization starting with an existing Pegasus-Large checkpoint. Figure 5 presents results for finetuning these models on the XSum and CNN/DailyMail datasets at various steps during pretraining.

Similar to our findings in Figure 4, our results with large models further confirm the advantages of the pretraining for planning; improvement over the Pegasus baseline were larger for the XSum dataset than for the CNN/DailyMail dataset. We achieved the best performance on both datasets at the 1m pretraining step. We use the Frost model at 1m pretraining step for our finetuning experiments.

Table 2 presents our final results from finetuning our large models on summarization datasets. For each dataset, we first report results from earlier work directly taken from corresponding papers; our results are in the bottom blocks for each dataset. We finetune our own Pegasus using the standard approach (document to summary). We report results from Frost (ecp; Entity Chain Planning), that is, Pegasus finetuned with the Frost objective (document to content plan and summary). Finally, we report on Frost (ecpp; Entity Chain Planning with Pretraining), our models both pretrained and finetuned with the Frost objective.

We find that even simply finetuning an existing Pegasus pretrained model to do content planning and summarization (as in Frost (ecp)) leads to improvements in entity specificity (EntF1) and the quality of entity plans in summaries (Entity Planning rouge), across all datasets. In fact, in some cases, better content plans lead to large improvements in summary-level rouge as well, for example, Frost (ecp) improve on Pegasus from 44.05/21.69/40.98 to 44.85/21.83/41.80 on rouge scores for CNN/DailyMail summaries.

The pretraining for content planning and generation in Frost (ecpp) further improves the entity chain quality for CNN/DailyMail, XSum and BillSum, and the summary-level rouge for CNN/DailyMail and XSum. Our Frost models establish new state-of-the-art rouge results on XSum. For CNN/DailyMail, we perform inferior to CTRLsum (He et al., 2020) and GSum (Dou et al., 2021) on rouge scores. However, we outperform CTRLsum on entity planning and specificity. Further discussion on comparing our methods to controlled summarization systems such as CTRLsum can be found in Section 5.5.

We demonstrate in two ways that planning with entity chains can be useful in mitigating hallucinations in summarization: data filtering and drop-prompt mechanism with entity chains. We focused on the XSum dataset for these experiments.⁸

During training Frost prepends reference summaries with their entity chains to better plan them. We leverage this to filter the dataset to only keep examples where summaries have fully extractive entity chains; an entity chain is fully extractive if all entities in it can be found in the input document. It’s notable that this filtered dataset will not contain novel entities in the summary targets. Frost models trained on this data will ground the summary generation to extracted entity chains while allowing abstraction for non-entity related generations. The resulting XSum dataset has 62.7k/3.5k/3.5k train/validation/test instances. We finetune our models from Table 2 on this filtered dataset and report their performance on the filtered test set (3.5k). We also evaluate them on the rest of the test set (7.8k) where reference entity chains are not fully extractive to their input documents.⁹ See Figure 6 for examples.

Frost decoders are trained to generate the summary s following the generation of its plan c. To improve faithfulness, we take the predictions from Frost (ecpp)¹⁰ and modify the generated plan c to c_drop by dropping entities (or parts of them) that are not found in the input document. We then prompt our decoder with c_drop to generate a new summary. We conduct this with both models, one trained on the full dataset and another on the filtered subset. We also report results for the oracle entity chain prompts c_oracle for a comparison.

Results are presented in Table 3. First of all, similar to our results in Table 2 on the full XSum test set, our results with the models trained on the original dataset further validate that the pretraining and finetuning with the content planning objective is equally useful for both test sets: one where entities are simply copied from the input documents and another where novel entities were inferred, to generate corresponding targets.

The data filtering using entity chains are particularly useful for test cases with extractive reference entity chains; models trained on the filtered data lead to much higher EntF1 (e.g., 65.97 vs 63.03, for Frost (ecpp)) and higher quality entity plans (entitly-level rouge of 66.43/30.00/62.48 vs 63.73/30.08/60.10 for Frost (ecpp)), without sacrificing the summary quality substantially (summary-level rouge of 44.87/22.05/36.79 vs 45.08/22.01/36.80 for Frost (ecpp)). Unsurprisingly, these models don’t do well on the test set with non-extractive reference entity chains, such examples were not seen during training.

The drop-prompt mechanism works particularly well for models trained on the full dataset and evaluated on the test set with extractive reference entity chains. For example, the entity planning rouge scores and EntF1 for Frost models improve from 63.73/30.08/60.10 to 67.10/30.19/63.58, and, 63.03 to 64.41, respectively. Interestingly, we do not observe a significant drop in the summary-level rouge scores (45.08/22.01/36.80 vs 44.97/21.97/36.77). Basically, our results demonstrate that when models are trained on the noisy data (with data divergence issues, common in summarization datasets (Dhingra et al., 2019; Maynez et al., 2020)), the drop-prompt mechanism is very effective in generating high quality summaries when entities need to be simply copied from the input documents.

The drop-prompt mechanism doesn’t help models when they are already trained on the filtered training set. Also this mechanism is counter intuitive to use for the test set where novel entities need to be inferred to generate targets, we observe drops for models irrespective of how they were trained, either using the full training set or the filtered set.

The planning objective in Frost itself does not ensure faithfulness; planning is done with the entities in the target, if the target is noisy, its plan may also be noisy (see Figure 6 for predicted entity chains with hallucinated entities). But the planning with the entity chains facilitates the data filtering and the drop-prompt mechanism, using entity chains. Table 4 presents our results assessing them for faithfulness. We randomly selected 50 documents for each of test sets (with extractive or non-extractive reference entity chains) and assessed their summaries from all 8 systems (except with c_oracle) from Table 3.

The data filtering using entity chains is extremely useful for improving faithfulness in summaries. We see improvements for all models trained on the filtered set (Table 4, bottom) compared to their counterparts trained on the full training set (Table 4, top), when evaluated using EntPrec and by humans for faithfulness. We don’t observe similar improvements in entailment. Contrary to findings in Maynez et al. (2020), our results suggest that the entailment scores are not a reliable indicator of faithfulness for documents with extractive reference entity chains.

The drop-prompt mechanism is also very powerful in improving faithfulness. Irrespective of which training data were used to train the models and which test sets they were evaluated on, we see improvements in entailment scores, EntPrec, and the human assessment of faithfulness across the board, except for a single case where the entailment score slightly drops from 0.465 to 0.453. Figure 6 demonstrates how we drop hallucinated entities ‘Liam’, ‘two’, and ‘Conor’ from entity chains to enforce models to generate faithful summaries grounded to the modified entity chains.

We achieve the best performance in terms of EntPrec and the human assessment of faithfulness when both the data filtering and the drop-prompt mechanism were used. We achieve 0.943 for EntPrec and 4.41 for faithfulness for the test set with extractive entity chains only, and 0.826 for EntPrec and 3.85 for faithfulness for the test set with non-extractive entity chains. Humans also found that the predictions from these models were the best in terms of overall summary quality.¹¹

Finally, we carried out pairwise comparisons for human assessments for faithfulness and overall summary quality for all models (using a one-way ANOVA with post-hoc Tukey HSD tests; p < 0.01). Interestingly, differences among all model pairs for both faithfulness and overall summary quality are insignificant when evaluated on the test set with extractive reference entity chains only. On the more challenging test set with non-extractive entity chains, Frost (ecpp) trained on the filtered training set and with the drop-prompt mechanism is significantly better than all other models except for (i) Frost (ecpp) trained on the filtered training set but without the drop-prompt mechanism and (ii) Frost (ecpp) trained on the original training set and with the drop-prompt mechanism, for both faithfulness and overall summary quality. All other pairwise differences are insignificant.

Frost provides a handle to easily control or manipulate content plan for predicted summaries. In Figure 6, we saw how we can use this to control entity-level hallucinations, but the strength of Frost models go beyond this. Figure 7 shows how Frost models can be effectively used in generating summaries with different entity focus by simply modifying the entity prompt. Future work will focus on how to leverage Frost for synthetic data generation for training improvements.

We compare the decoding strategy in Frost to encoder-guided control summarization systems (He et al., 2020; Dou et al., 2021). Table 5 presents our results. In particular, we report on CTRLsum (He et al., 2020); first, a keyword extraction system (BERT-based sequence tagger) is used to extract keywords (k) from the input document d, and then, the extracted keywords are encoded along with the input document (as k;d) to generate the summary. We also finetune Pegasus where the entity chain (c) is encoded along with the input document d (as d;c or c;d). We only report the oracle results (with c_oracle) for these models; like CTRLsum, generating an entity chain c during inference will require training an additional entity chain generator, which is out of scope of this paper.

There are several advantages of Frost-style decoding. Unlike encoder-guided control summarization systems, Frost models can be used in a usual way to generate summaries for input documents without relying on external systems to augment them during inference. Additionally, users can modify the generated entity prompts or provide their desired entity prompts to control the generated summaries (see Section 5.3 and 5.4). To the best of our knowledge, Frost is the first decoder-prompted control summarization model.

Our results in Table 5 show that the prompting the decoder with entity prompts are more effective in generating high-quality and grounded summaries compared to when entity prompts are encoded with the input, especially for abstractive summaries; both Pegasus versions (d;c_oracle and c_oracle;d) perform inferior to Frost (⁠$d→coracle;s$⁠) in terms of summary-level rouge, entity-level rouge and EntF1 on the XSum dataset. CTRLsum (⁠$koracle;d→s$⁠) achieves better rouge scores than Frost (⁠$d→coracle;s$⁠) on CNN/DailyMail (64.72/40.56/61.02 vs 61.85/38.95/59.00). This is not surprising as the oracle keywords k_oracle in CTRLSum retain most words from the summary found in the source document; as such due to the extractive nature of CNN/DailyMail summaries, k_oracle tend to be closer to the surface forms of the reference summaries (rouge scores between k_oracle and reference summaries are 52.70/17.37/44.37). The difference in rouge scores narrows down between CTRLsum and Frost to 45.65/22.35/42.50 vs 45.11/22.11/42.01 with automatically extracted keywords. Frost outperforms CTRLsum on entity planning and specificity. It is not clear how well CTRLsum’s keyword extraction system will generalize to more abstractive datasets where words in reference summaries are often not found in the source document.

Finally, our Frost models work as a better microplanner and produce concise summaries than those generated by systems without doing content planning or when its done on the input side. For example, the average lengths of CNN/DailyMail test summaries are 68.65, 74.46, 65.75 and 60.70 tokens for Pegasus, CTRLSum, Frost, and humans, respectively. See example predictions comparing CTRLSum and Frost in Figure 8.

In this work, we proposed the use of entity chains, both during pretraining and finetuning, to better plan and ground the generation of abstractive summaries. Our approach achieves state-of-the-art rouge results in XSum and SAMSum, and competitive results in CNN/Dailymail. Compared to other guided summarization models, CTRLSum and GSum, which perform slightly better in CNN/Dailymail, our approach is drastically simpler to implement, trains by augmenting the targets only, utilizes no additional parameters than the baseline Pegasus model, and does not rely on any external systems to augment the inputs during inference. We further demonstrate that by modifying the entity chain prompts, we can easily control hallucinations in summaries.

We thank the reviewers and the action editor for their valuable feedback. We would like to thank Rahul Aralikatte, Ankur Parikh, and Slav Petrov for their insightful comments. Many thanks also to Ashwin Kakarla and his team for help with the human evaluations.

Tables 6 and 7 present hyperparameters used for pretraining and finetuning Pegasus and Frost base and large sized models.

Figures 9 and 10 show the exact instructions and the template used by our annotators for faithfulness and overall summary quality, respectively.