Neural Data-to-Text Generation Based on Small Datasets: Comparing the Added Value of Two Semi-Supervised Learning Approaches on Top of a Large Language Model

Neural NLG methods are notoriously data hungry, and rely on large-scale datasets that typically require large amounts of effort and resources to construct (Gkatzia 2016). The fact that such datasets are rare and difficult to develop creates a so-called data bottleneck (Oraby et al. 2019). Due to the lack of large datasets, many neural NLG approaches rely on relatively small datasets, which not only affects output quality, but also output diversity (Holtzman et al. 2020).

One of the NLG subtasks that especially suffers from the consequences of small-scale datasets is data-to-text generation: The task of producing adequate, fluent and natural language text from non-linguistic structured data (Gatt and Krahmer 2018). (Supervised) neural data-to-text NLG involves the collection of parallel data–text datasets, aligning data and linguistic realizations of these data. However, collecting these datasets is difficult because sets of texts and corresponding data are not a common natural occurrence (Shimorina, Khasanova, and Gardent 2019). On the other hand, unpaired texts and data are significantly more common and easily collected (Qader, Portet, and Labbé 2019). While these unpaired texts and data do not lend themselves to supervised data-to-text generation, they can be utilized by means of semi-supervised learning, the process of training a model on existing data–text pairings, and having this model create more synthetic pairings for the training set (see also Figure 1).

Although more data generally leads to better performing machine learning systems, it is unclear to what extent a system benefits from including (possibly imperfect) synthetic data. Caution is advised when working with such data, as the inclusion of synthetic data may reinforce a model’s mistakes (He et al. 2020). This issue becomes further exacerbated in the context of data-to-text generation, where neural models can be prone to issues such as hallucination, repetition, and omission of data (Faille, Gatt, and Gardent 2020). However, previous data-to-text generation studies suggest that semi-supervised learning increases output quality compared with a supervised approach, especially when the labeled dataset size is small (e.g., Qader, Portet, and Labbé 2019; Schmitt et al. 2020; Su, Huang, and Chen 2020; Tseng et al. 2020; Chang, Demberg, and Marin 2021). Besides improved output quality, adding more training examples using semi-supervised learning also might increase the language diversity of the output.

Recent developments in large-scale language models, utilizing the Transformers architecture (Vaswani et al. 2017), may offer an alternative solution to the data bottleneck. They leverage data from various domains to supplement the in-domain information that is available (Sun et al. 2020). These language models have also been found to have a beneficial impact on output quality for the data-to-text task, just like semi-supervised learning (e.g., Kale and Rastogi 2020). Both of these approaches aim to improve the data-to-text training model by providing extra information in addition to the information that is present in the training set as an enriched training set. Therefore, it is conceivable that the massive amounts of information already incorporated in a language model makes the use of semi-supervised learning redundant.

However, studies have also shown that the beneficial effect of a language model decreases when a model is overfit too much during finetuning (Greco et al. 2019). Overfitting is also more likely to occur with small datasets. The fact that semi-supervised learning increases the dataset size may therefore help against overfitting. If so, language models and semi-supervised learning would be complementary approaches that might lead to better performance when used in conjunction. However, not much is known about the effect that this combination of language-models with semi-supervised learning has in a data-to-text setting. Furthermore, provided that the addition of semi-supervised learning is beneficial, the type of semi-supervision approach that leads to the best results is currently unknown, as experimental comparisons of this are scarce as well.

This study will investigate when and how semi-supervised learning affects the output diversity and quality when used in combination with language models for data-to-text generation. Two different semi-supervised learning approaches are investigated—both utilizing pretrained Transformers models (Vaswani et al. 2017)—and the impact that these approaches may have on the output quality and diversity of a neural data-to-text system with a language model. The semi-supervised learning approaches used in this study are: (I) a data augmentation approach, where several variants of a training text are generated by replacing certain words with synonyms or semantically similar words, and (II) a pseudo-label approach, where unlabeled texts are given data labels by an information extraction (semantic parsing) model trained on the existing labeled training data. The synthetic data–text pairs obtained via these two approaches are then added to the original training set in a neural data-to-text system to generate new texts.

Even though a handful of recent papers have started exploring the benefits of semi-supervised learning for data-to-text generation, this is the first study presenting a detailed, large scale analysis (1) of different corpora, with vastly different characteristics, in two different languages (vs. previous approaches that mostly focused on only one corpus), (2) that systematically compares different methods for semi-supervised learning, also in combination with pretrained language models (vs. previous approaches that mostly did not incorporate pretrained language models in the semi-supervised learning approaches, nor the baseline), and (3) that performs an exhaustive evaluation of the different methods, by combining automatic analyses with a human evaluation, error analysis, and qualitative analysis, in line with recent best practices for evaluation (vs. previous approaches that mostly focus on automatic metrics or only conduct a limited human evaluation).

Based on previous studies, we formulated several hypotheses and a more exploratory research question before conducting the study. All these hypotheses and research questions have also been preregistered before conducting the study at https://aspredicted.org/in665.pdf, following the advice given by van Miltenburg, van der Lee, and Krahmer (2021). In this subsection, we will state these hypotheses and research questions, followed by the rationale for the expected effects.

H1.Extending the training set with semi-supervised learning increases the output quality of a neural data-to-text system with a language model (compared to a data-to-text system with a language model only trained on the base training set).

Previous studies have consistently found that semi-supervised learning leads to improvements in output quality (e.g., Kulhánek et al. 2021; Riabi et al. 2021; Tandon et al. 2018; Alberti et al. 2019; Chang et al. 2021; Kedzie and McKeown 2019). Additionally, Sun et al. (2020) examined the dynamics between semi-supervised learning and language models in a text classification setting and found that a combination of the two approaches led to the highest classification scores.

H2.Extending the training set using semi-supervised learning increases the output diversity of a neural data-to-text system with a language model (compared to a data-to-text system with a language model only trained on the base training set).

Finetuning a text generation system with language models on small or non-diverse training data may lead to limited diversity in the output, even though language models themselves are trained on enormous amounts of text. This has to do with the known propensity for catastrophic forgetting that neural networks display (Greco et al. 2019): The model is overfit too tightly during finetuning, which leads to “forgetting” about the capabilities of language modeling. Therefore, extending the finetuning dataset with more various language using semi-supervised learning could have a positive impact on diversity, which has also been found in a study by Kulhánek et al. (2021).

RQ1.Do the data augmentation and pseudo-label approaches differ in terms of output quality and output diversity when used as semi-supervised learning approaches in a neural data-to-text system with a language model?

Although different semi-supervised learning approaches have shown their potential for various NLP/NLG tasks, not much is known about which semi-supervised learning approaches are the most effective (Sun et al. 2020).

H3.The beneficial effect of semi-supervised learning for a neural data-to-text system with a language model on output quality is bigger for a small-scale dataset (CACAPO) than for a large-scale dataset (WebNLG and E2E).

For datasets, we make a distinction between crowdsourced on the one hand (e.g., WebNLG, E2E, ToTTo; Novikova, Dušek, and Rieser 2017; Gardent et al. 2017a, 2017b; Parikh et al. 2020), and datasets that are created from “naturally occurring” human-written texts on the other (e.g., YelpNLG, RotoWire, CACAPO; Oraby et al. 2019; Wiseman, Shieber, and Rush 2017; van der Lee et al. 2020). For the construction of a crowdsourced dataset, crowdsource workers write corresponding texts for a given set of meaning representations. This approach is reasonable for the construction of a large-scale dataset, provided that time and resources are not an issue.

However, the very procedure of using crowdsource workers to verbalize a set of meaning representations without any given context ensures that texts are mostly focused on high fidelity, while sacrificing on criteria like fluency and enjoyability (van der Lee et al. 2020). This means that output from systems trained on crowdsourced datasets is less likely to contain diverse language. Alternatively, it is possible to construct a dataset with texts written in real-world scenarios, rather than lab-setting verbalizations by crowdsource workers. This can be done by collecting texts and corresponding data on a large scale, without having detail-level alignment information (as is done in Wiseman, Shieber, and Rush 2017; Wang 2019; Puduppully, Dong, and Lapata 2019), or by manually extracting aligned data from texts (as is done in Oraby et al. 2019; van der Lee et al. 2020). While such an approach is likely to facilitate more diverse and fluent language, the difficulty of this alignment task makes large-scale dataset collection a daunting endeavor.

In the current study we will compare the two most widely used data-to-text datasets: E2E (Novikova, Dušek, and Rieser 2017) and WebNLG (Gardent et al. 2017a, 2017b), which are both large-scale and crowdsource-based, and CACAPO (van der Lee et al. 2020), which is smaller-scale and based on real-world texts, in two languages (viz. English and Dutch). Previous studies have suggested that the benefits of semi-supervised learning are greater in more low-resource scenarios (Chang, Demberg, and Marin 2021).

H4.The beneficial effect of semi-supervised learning for a neural data-to-text system with a language model on output diversity is bigger for a crowdsourced dataset (WebNLG and E2E) compared to a dataset based on real-word texts (CACAPO).

Focusing on diversity, datasets based on real-world texts generally contain more diverse language than crowdsourced datasets (van der Lee et al. 2020). This implies that crowdsourced datasets have more to gain from semi-supervised learning approaches’ potential to introduce more language diversity to the training set of crowdsourced datasets.

H5.The beneficial effect of semi-supervised learning is greater when trained on a Dutch dataset (CACAPO Dutch) than an English dataset (WebNLG, E2E, and CACAPO English) for a neural data-to-text system with a pretrained language model.

It is a well-known fact that the majority of NLP developments are focused on English, with many other languages lagging behind in terms of support (Bender et al. 2021; Riabi et al. 2021). This is also the case for language models. Although such language models exist for other languages, and multilingual variants of state-of-the-art models exist (e.g., mBert, mT5; Devlin et al. 2019; Xue et al. 2021), the size of these models is generally much smaller (Bender et al. 2021), and they are oftentimes missing functionality. For instance, mT5 was not pretrained on downstream tasks like T5 was (Xue et al. 2021). This means that the benefits of language models in data-to-text generation is likely smaller for underrepresented languages, which in turn might mean that the beneficial effects of semi-supervised learning is greater, especially when we take into consideration that the size of non-English datasets is generally smaller as well (Riabi et al. 2021).

The goal of semi-supervised learning is to train a model (partially) on synthetic data (as opposed to data created by humans), which may lead to a better trained machine learning model, and hence improved performance. Figure 1 gives a schematic overview of semi-supervised learning and how it differs from supervised learning, which is the standard for neural NLG. This approach has steadily grown in popularity with the rise of data-hungry neural models, and is considered especially useful when the (labeled) training set is small-scale. Within the NLG domain, we have seen applications of this approach in, for instance, question answering (e.g., Alberti et al. 2019; Riabi et al. 2021), and text simplification (e.g., Surya et al. 2019; Zhao et al. 2020).

The semi-supervised approach has also gained traction in the context of data-to-text generation. This is mostly in the form of joint learning systems, where an NLG system (that converts meaning representations into text), and a Natural Language Understanding system (that converts text into meaning representations) are feeding each other more synthetic data in a loop. Such an architecture allows for both unlabeled texts as well as meaning representations without aligned text to be included into the training data. Some studies suggest that the use of a joint learning system led to improvements on various metrics compared to a supervised NLG system (e.g., Qader, Portet, and Labbé 2019; Schmitt et al. 2020; Su, Huang, and Chen 2020; Tseng et al. 2020; Chang, Demberg, and Marin 2021).

The architecture of the current research differs from these previous studies as it utilizes unlabeled/unaligned data in a non-joint way. This approach is based on the assumption that it is easier to extract information from a text than to generate text that accurately represents information (Wiseman, Shieber, and Rush 2017). It should also be noted that previous data-to-text studies using semi-supervised approaches partition a segment of the dataset and detach the data from the texts to create unlabeled data, or they use value swapping (i.e., pairing each key with a randomly sampled value collected from the set of all data samples to obtain new combinations of key-value pairs) to create extra unlabeled data.

Our approach tries to more closely emulate the application of this task in a real-world setting by collecting and utilizing unaligned texts that are not found in the datasets. Furthermore, none of the previous studies used language models in the architecture of their data-to-text generation system and only one previous study utilized these language models for semi-supervised learning (Chang et al. 2021).

Transformers-based language models have particularly shown their viability for generation tasks that involve meaning manipulation (e.g., summarization, text simplification, and question answering), but studies also suggest that the inclusion of Transformers-based language models can lead to improvements in output quality for the data-to-text generation task (Chen et al. 2020; Kale and Rastogi 2020; Mager et al. 2020; Ribeiro et al. 2021). Furthermore, Transformers-based language models have been found to perform well on very small datasets, with examples existing of few-shot, one-shot, or even zero-shot learning (Brown et al. 2020). This impressive performance might suggest that including language models in the architecture of an NLG system might make semi-supervised learning approaches redundant. However, it should be noted that language models and most semi-supervised learning approaches utilize different types of data. While language models leverage data from an immense variety of different domains, semi-supervised learning approaches are generally focused on using in-domain data (Sun et al. 2020).

Therefore, a combination of language models and semi-supervised learning approaches might enhance performance, rather than canceling each other’s improvements out. Sun et al. (2020) find support for this notion in a text classification context. They found that the largest performance gain was achieved when the two were combined (Sun et al. 2020). The authors also compared different semi-supervised learning approaches (in-domain pretraining and pseudo-labeling) and found performance differences between the two. Similarly, the current study also compares the performance of two semi-supervised learning approaches, but in a data-to-text generation context: pseudo-labeling and data augmentation.

One of the most common semi-supervised learning approaches is the pseudo-labeling approach, where unlabeled data is assigned labels by a model, thus forming a large labeled dataset that can be used to train a model. In the context of NLP, this task is equal to information extraction (also known as semantic parsing, or natural language understanding), where a meaning representation is parsed from a text. Most of the existing semi-supervised NLG systems have utilized information extraction for the creation of synthetic training data (e.g., Qader, Portet, and Labbé 2019; Schmitt et al. 2020; Su, Huang, and Chen 2020; Tseng et al. 2020; Chang, Demberg, and Marin 2021).

However, most of these studies also apply pseudo-labeling without utilizing any language model. Besides their suitability for various generation tasks, Transformers-based language models have also shown their potential for information extraction. For example, various authors have applied BERT-based information extraction successfully on small datasets (e.g., Nguyen et al. 2019; Zhang et al. 2020a), and one of the best performers on the semantic parsing subtask of the WebNLG+ Challenge 2020 (Castro Ferreira et al. 2020) was a parser that used T5 as a pretrained model (Agarwal et al. 2020). Building on these previous findings, the current study also utilizes an approach to pseudo-labeling that includes a pretrained model.

Data augmentation generally refers to all strategies that increase training examples without explicitly collecting new data (Feng et al. 2021). This can be done for instance by adding slightly edited copies of existing texts, which has been applied in the data-to-text generation context through methods such as (back) translation (Kulhánek et al. 2021; Riabi et al. 2021).

More common, however, is data augmentation in the form of self-training, where a first version of a data-to-text model generates new texts from inputs, which are then fed to the model for further training (Heidari et al. 2021; He et al. 2020). In a data-to-text generation context, this has specifically been done through text generation from modified meaning representations (e.g., Tandon et al. 2018; Alberti et al. 2019; Chang et al. 2021), by noise injection (Kedzie and McKeown 2019), or by simply generating texts from unpaired inputs (e.g., Heidari et al. 2021; Jolly et al. 2022; Mehta et al. 2022). With few exceptions (i.e., Jolly et al. 2022), these approaches were all found to increase output quality.

A less common way of data augmentation in data-to-text generation is by using synonym replacement and text editing, which has effectively been applied for text classification (e.g., Zhang, Zhao, and LeCun 2015), and hate speech detection (Emmery et al. 2022; Rizos, Hemker, and Schuller 2019). However, recent advances in learning-based quality estimation metrics, such as RUSE (Shimanaka, Kajiwara, and Komachi 2018), BERTScore (Zhang et al. 2020b), MoverScore (Zhao et al. 2019), and BLEURT (Sellam, Das, and Parikh 2020) try to gauge the semantic similarity of generated sentences compared to a gold standard using language models. The ability of these metrics to detect synonyms and semantically similar language does illustrate the viability of synonym replacement in (data-to-text) NLG as well, as instilled knowledge of semantically similar words and phrases is the most important part of data augmentation based on synonym replacement. In the current study, we will further investigate the potential for (the synonym/semantically similar replacement approach for) data augmentation using language models as a semi-supervised learning approach in data-to-text generation.

For the data augmentation approach, it is beneficial to be able to locate the exact position where data was verbalized in the text, so that augmentations in the text that have to do with the data could easily be traced back and changed in the data as well. This ensures that augmented variants of texts also align with its data counterpart. Therefore, datasets were chosen for this experiment that included such enriched information. These are: (enriched) E2E (Castro Ferreira et al. 2021), (enriched) WebNLG (Castro Ferreira et al. 2018), and CACAPO (van der Lee et al. 2020). We used the original train/development/test splits for these corpora. Specific characteristics of the datasets are discussed in more detail below.

It should also be noted that we differentiate between domains for the CACAPO and WebNLG dataset (E2E is only one domain), as we believe that treating them separately will not only result in higher performance, but also provide richer and more detailed information about the performance of the various methods. Domains in CACAPO and WebNLG are inherently different due to imbalances that exist in the data (mostly in WebNLG) and due to the very nature of the reports. More specifically: The differences in the richness of information they provide, and the complexity and diversity of the language that is used. To fully capture the effects that these domain differences have on the performance of different approaches, it is necessary to look at the sub-corpora.

Furthermore, we added synthetic data to the original dataset in various quantities. This was done to exploratively study the effects of semi-supervised learning in a more detailed fashion. It could, for instance, reveal that there is a saturation point where adding more synthetic data stops improving performance, or that performance decreases when more synthetic data is added which indicates cascading of errors. Sizes and statistics are described in more detail below.

E2E is focused on the restaurant domain and contains English verbalizations of data, which were collected using crowdsourcing. The data for this dataset is stored in a key-value format, similar to CACAPO. The dataset is split in a training, development, and test ratio of 76.5%/8.5%/15%, respectively. Of the three datasets in this study, E2E is the largest in terms of sheer size: Table 1 summarizes the basic statistics for E2E (and the other corpora used in this study). E2E contains 42,061 instances (i.e., aligned data–text pairs), and 840,760 tokens. Furthermore, it contains 4,862 unique meaning representations (i.e., data elements), less than the other two corpora, which suggests that this dataset contains more repetition compared with WebNLG and CACAPO.

As there is no information available on the origins of the data used for E2E, it is difficult to collect new data or find comparable restaurant descriptions online. Therefore, we used E2E+ (Roberti et al. 2020) as extra data for the pseudo-labeling approach. E2E+ is a modified version of E2E where all slot data is replaced with comparable data. For instance, food data is replaced using the adjectival forms of countries and nations found on Wikipedia, and name and near are replaced with New York restaurant names found in the Entree dataset (Burke, Hammond, and Yound 1997) (e.g., Blue Spice serves highly rated Chinese food. becomes El Charro serves highly rated Timorese food). To investigate the effect of the pseudo-labeling approach, four sizes of aligned data–text information were added to the original training set: Small, medium, large, extra large. These contained 12.5%, 25%, 50%, and 100% of the E2E+ data, respectively, in the case of the pseudo-labeling approach.

WebNLG is collected in a similar crowdsourced manner as E2E and is derived from DBPedia properties. These properties are different from E2E and CACAPO data as they are not stored in a key-value format, but as SVO-triples (subject-verb-object). Each of these properties is related to a particular category in DBPedia. For the enriched WebNLG dataset, these domains are: Airport, Astronaut, Building, City, ComicsCharacter, Food, Monument, SportsTeam, University, and WrittenWork. While the dataset is smaller than E2E in terms of tokens and instances (24,404 instances; 349,712 tokens), it does seem more varied in its composition, as evidenced by the number of unique meaning representations (10,672) (see Table 1). Furthermore, the dataset is split by a 60%/20%/20% training, development, and test ratio.

Following Montella et al. (2020), we collected Wikipedia texts as extra data for the pseudo-labeling approach for WebNLG. These texts are similar in nature, as Wikipedia pages are generally well-connected to the DBPedia variant of the page. Furthermore, Wikipedia texts are freely available and relatively easy to collect. For each of the DBPedia categories in the WebNLG dataset, we searched for similar overview pages on Wikipedia, and then scraped all the pages in the overview or in the subcategories of the overview.¹ Then, the summary (i.e., the first paragraph of the article) was taken from each page, split on a sentence-level, labeled, and added as extra data. The training set was extended with the sentences of 125, 250, 500, and 1,000 summaries for, respectively, small, medium, large, and extra large. Because the pseudo-labeling data was split on a sentence-level, the original training set was also split on a sentence level to ensure consistency between the original training data and the input derived from the pseudo-labeling approach.

The CACAPO dataset (van der Lee et al. 2020) contains texts from the Sports, Weather, Stocks, and Incidents domain for both Dutch and English. Each domain contains information for 200 texts (1,600 texts total), paired with manually annotated data for each sentence in a key-value format. It is split up in a 76.5%/8.5%/15% training, development, and test ratio, similar to E2E.

Besides language differences, there are also topical differences between the English and Dutch part of the dataset: The Weather and Stocks report are relatively similar in their content, but the Dutch version of the Sports domain contains soccer reports, whereas the English version contains baseball reports (based on Puduppully, Dong, and Lapata 2019). Similarly, the Dutch Incidents domain contains reports about traffic incidents (from Hendriks 2019), while the English Incidents domain contains reports about firearm incidents (see van der Lee et al. 2020, for a detailed description). Size-wise, both the Dutch and English datasets are the smallest in this study in terms of instances (Dutch: 7,367, English: 7,923) and tokens (Dutch: 110,391, English: 153,663) (see Table 1). The large number of meaning representations (Dutch: 6,590, English: 7,613) indicates a relatively large variation for its size.

Unlabeled texts for the pseudo-labeling approach were scraped using the same text collection methods as were used for the CACAPO dataset. This means that human-written texts were collected from the same selection of Web sites as were used for CACAPO. Furthermore, the texts were collected using an automatic scraper, or a tool that made saving texts in a correct format as effortless as possible, as was also done in the construction of CACAPO. Similar to WebNLG, the small version of the training set was extended with the sentences of 125 articles for small, 250 articles for medium, 500 articles for large, and 1,000 articles for extra large.

For data augmentation, we use lexical substitution (McCarthy and Navigli 2007); that is, for specific words in the input (the target words) we determine multiple alternatives that are semantically similar (the substitution candidates). For this purpose, we use Emmery, Kádár, and Chrupała’s (2021) implementation of Zhou et al.’s (2019) work. Under the framework of masked language modeling, to predict synonyms rather than any word fitting a particular (masked) position, Zhou et al. (2019) proposed using Dropout (Srivastava et al. 2014). The pipeline of this architecture is displayed in Figure 2.

Instead of masking a selected word in the text, Dropout is applied to the BERT-internal embedding of that word. The intuition is that, rather than BERT predicting identical words when the original word’s embedding is passed, the partly-zeroed embedding results produces synonyms instead. These are then the substitution candidates, which we contextually re-rank using a similarity score.²

Candidates are removed if they do not match certain criteria: Their similarity scores should be > 0.9, and should not be punctuation or single characters, UNK tokens, plurals or capitalized versions of, or equal to the target word, subwords, or already exist in the sentence. BERT-large (Devlin et al. 2019) was used to generate the candidates for English, BERTje (de Vries et al. 2019) for Dutch, and Dropout was set to 0.2.

For the target words, we chose all nouns, adjectives, adverbs, and numerals—tagged using SpaCy (Honnibal and Montani 2017). Similar to Emmery et al. (2022), we fill each position with a candidate simultaneously (i.e., using the highest ranked candidates for each target word to produce the first augmented instance, and so on; e.g., “What will the weather be like this afternoon in Preston?”→“What will the air be like this evening in Manchester?”). We repeat this step for a maximum of twenty instances. If target words do not have up to the maximum amount of substitution candidates, they are left as the original words instead. The top 1 (small), 2 (medium), 5 (large), and 10 (extra large) instances of each text, based on their BERT similarity score with respect to the original sentence, were then added to the training datasets. As previously noted, enriched versions of corpora were used for data augmentation to ensure that augmentations were also applied to the aligned data.

We acknowledge the difficulty of measuring the performance of data augmentation using automatic metrics, since most metrics are based on a comparison to a gold standard. Furthermore, language model-based semantic distance metrics (such as BERTScore and BLEURT) are very similar in nature compared to the data augmentation approach used in this study, which might make their scores more akin to a manipulation check rather than an accurate reflection of semantic similarity between an augmented sentence and its original. Still, performing an evaluation using these metrics offers novel information as it measures sentence-level semantic consistency, which has not been measured in full during the data augmentation process.

Therefore, we calculated the average BLEU (Papineni et al. 2002), BLEURT (Sellam, Das, and Parikh 2020), and BERTScore (Zhang et al. 2020b) scores of the augmented texts compared to their original. A lower BLEU score (a straightforward metric that measures text-overlap between a candidate and reference) and higher performance on BLEURT and BERTScore (metrics that aim to measure semantic similarity between a candidate and reference) might suggest that texts have been augmented fundamentally, while still conveying a semantically similar message. Furthermore, we used LanguageTool³ to calculate the difference in grammatical errors compared to the original sentences, as an indicator for (relative) grammatical correctness.

The results indicate that the data augmentation was generally effective. The BLEU scores are mostly around 10–20 for all domains, although a few exceptions exist. These low BLEU scores suggest that a large chunk of the original sentences were modified, meaning that the training data became more varied. We also see that the BLEURT and BERTScore numbers are higher than BLEU for almost every domain, with scores in the 40–50 range for all domains. The BERTScores for the Dutch domains are an exception, which rise above 80 due to the fact that rescaling with a baseline is not possible for this language.

Nevertheless, these scores seem to indicate that the semantic similarity to the original text is kept relatively intact. Finally, the difference in grammatical errors is small, with generally only 0.1 to 0.2 more errors being found in the augmented texts compared to the original texts. This suggests that data augmentation adds few new grammatical errors to the texts. Thus, overall, we can see these scores as an indicator that the data augmentation approach indeed manages to modify a sentence fundamentally, while still keeping the text relatively semantically similar to the original and relatively error-free (see Table 2).

Similar to Schmitt et al. (2020), we framed the pseudo-labeling task as a text-to-text translation task, as this approach could handle the differences in data formats between all three datasets most effortlessly and effectively (compared with, for instance, span labeling, or extractive question answering). While the most straightforward text-to-text translation purpose is to translate a text from, for instance, English to German, text-to-text translation can actually be used effectively for a multitude of natural language processing tasks, as most of these involve conversion of one text format into another. T5 (Raffel et al. 2020) was developed with this purpose in mind.

This architecture, also known as Text-to-Text Transfer Transformer (T5), is a large pretrained language model resulting from an empirical survey to determine which transfer learning techniques work best. Different from classification language models such as BERT, T5 works as a unified text-to-text-approach where all the NLP tasks are reframed such that its inputs and outputs are strings. This text-to-text framework is a multi-task one, sharing the parameters, loss function, and hyperparameters on any NLP task, including machine translation, document summarization, question answering, and classification tasks (e.g., sentiment analysis). To set the desired task for the model, a prefix needs to be inserted in the input such as “translate English to German” for the machine translation task or “summarize” for the summarization one.

In this case, we “translated” a “data language” to Dutch or English using T5-large (Raffel et al. 2020) for pretraining of the English pseudo-label model, and mT5-large (Xue et al. 2021) for pretraining of the Dutch pseudo-labeling model (following Agarwal et al. 2020). This was done using run_translation.py from https://github.com/huggingface/transformers (Wolf et al. 2019) with 30 epochs and a batch size of 8. T5 and mT5 were further finetuned on the original training and development set of the CACAPO, E2E, and WebNLG datasets and applied to the test set to calculate performance. Furthermore, the trained models were applied to the unlabeled new texts that were collected to extend the training set.⁴

More specifically, we converted the data into a structured string format that follows the data structure of the dataset (see Figure 3) using “translate Dutch [resp., English] to Data: [...]” as the prefix command. For the output, this string format was then converted back into structured data using a simple rule-based script. The pipeline of this architecture is displayed in Figure 4.

We evaluated the performance of the pseudo-labeling approach by calculating the precision, recall, and micro-averaged F1 score on the development and test sets of all datasets. While we believe that these measures give a robust indication of the labeling quality, it should be noted that the model might not generalize well to the unlabeled texts, especially when the unlabeled texts are highly dissimilar from the texts seen in training (for instance, the pseudo-labeling model trained on E2E showed a considerable drop-off on the E2E+ data.⁵)

Overall, the scores indicate that this pseudo-labeling approach performs well, with F1 scores well above the 70s for CACAPO and even in the high 80s and 90s for WebNLG and E2E (see Table 3). Two notable exceptions are the City and Monument domains for WebNLG, which achieve much lower scores than other domains. This is likely caused by imbalanced data in the WebNLG dataset, which is especially prevalent in the City and Monument domain.

The data-to-text approach utilized in this study was a neural end-to-end architecture where a set of input data is directly converted into English or Dutch text. This was done using Any2Some,⁶ which uses language models from the HuggingFace API (Wolf et al. 2019) to perform data-to-text generation, while offering advantages such as automatically clustering verbalizations based on the same data. Similar to the pseudo-labeling step, we used T5-large (Raffel et al. 2020) for the data-to-text conversion step as well. This time, using the text-to-text nature of the language model to perform data-to-text generation using “Verbalize: [...]” as the prefix command.

As mentioned previously, T5 has been developed as an approach capable of handling a multitude of Natural Language Processing tasks where the inputs and outputs are reframed as strings. For this system, the input and output used in the pseudo-labeling step was essentially reversed: The data was again converted to a structured string format but this time used as input, with English or Dutch text serving as output. Previous research has suggested that T5 is a capable language model for the data-to-text generation task (Kale and Rastogi 2020).

The model was finetuned on all individual domains for 16 epochs with a learning rate of 1e-5, early stopping of patience 5, and batch size of 2. Input and output strings were trimmed to a maximum size of 180 sub-tokens. For the Dutch CACAPO domains we used mT5-large, with the same hyperparameters except for 50 epochs. More epochs were necessary for this model to be properly trained, as mT5 was not trained on downstream tasks. Some examples of the input and output of the data-to-text generation system for each dataset and semi-supervised learning method can be found in Table A.1.

The goal of this evaluation study was to investigate the contribution of the semi-supervised learning methods in data-to-text NLG. To investigate this, the evaluation study consisted of three parts: an automatic evaluation, a quantitative human evaluation, and an error analysis.⁷ We aimed to follow the best practice guidelines as described in van der Lee et al. (2021) as much as possible in the setup and reporting of the evaluation study.

For the automatic evaluation, multiple metrics were used to estimate output quality and output diversity. The quantitative human evaluation experiment measured aspects of text quality to further determine the performance of the different semi-supervised approaches relative to each other, and finally an error analysis was performed on the 15 worst scoring sentences per dataset × semi-supervised learning approach combination to investigate the shortcomings and challenges for each semi-supervised learning approach.

The performance of the three types of semi-supervised learning that were investigated in this research (no extension, data augmentation, and pseudo-labeling) was first tested for all domains in the CACAPO, E2E, and WebNLG datasets using automatic metrics that measure text quality and diversity. The text quality metrics served as a first test for H1, H3, H5, and RQ1. The text quality metrics employed in this study are displayed in Table 4.

Furthermore, we use the diversity metrics based on van Miltenburg, Elliott, and Vossen (2018). These metrics are used to test H2 and H4, as they provide an objective and complete image of the diversity in the output of the systems, which cannot be measured as accurately with sentence/phrase-level human evaluation. The diversity metrics used in this study are described in Table 5.

Participants of this study were recruited via Prolific, a crowdsourcing platform. For participation, participants received $4.80 (the recommended amount according to the platform). In total, 193 people participated in the study, which was divided up in a Dutch version and an English version. In the recruitment phase, only participants who were native Dutch located in the Netherlands were recruited for the Dutch version, and native English speakers located in the United States for the English version. This resulted in 41 participants in the Dutch version, of which the majority were men (56%) between the ages of 18 and 34 years (90%). Furthermore, the majority of the Dutch sample had a university (of applied sciences) degree (79%). For the English version, 152 people participated in the study. The majority were women (64%), roughly half of the participants were between the ages of 18 and 34 years (48%), and the majority had attended or completed college (87%).

To ensure that we captured the variety found in all datasets and among all semi-supervised learning approaches, we measured the text quality of outputs from the 3 investigated semi-supervised learning approaches on all 19 domains in the datasets we used (CACAPO: 8, E2E: 1, WebNLG: 10). We randomly sampled a total of 40 items per semi-supervised learning approach–domain combination, leading to a total of 19 × 3 × 40 = 2,280 trials. Each trial was judged a total of 5 times to obtain a stable judgment of the trial.

Each participant was randomly assigned to a dataset domain. Dutch-speaking participants were randomly assigned to 1 of 4 domains (the four Dutch CACAPO domains), while English-speaking participants were randomly assigned to 1 of the other 15 (English) domains. Furthermore, each dataset domain had 2 versions, with each version containing 60 outputs total from the systems trained on the XL data (20 per semi-supervised learning approach) that were all not present in the other version. This number of outputs was chosen to ensure that the sample contained enough variety to be representative of the variation in the full dataset, while the number of stimuli presented to individual participants was still manageable for them. Participants were randomly assigned to 1 of the 2 versions. The 60 outputs were presented in random order to compensate for potential order- or fatigue effects.

A survey was created using the Qualtrics platform. First, a general introduction of the experiment and a consent form was given to the participants. After consenting to participate in the research, the participants were given detailed instructions about the experiment they were about to participate in. These instructions included guidelines on how to read the data input and the output texts, and how to rate said output texts. Furthermore, definitions were given for the scales they had to rate, and examples were given about good output texts and bad output texts. Instructions, guidelines, definitions, examples, as well as the questions themselves (as shown below), were translated to Dutch for the Dutch version of the evaluation to ensure that monolingual Dutch participants were able to comprehend the contents.

After these instructions, participants were asked to provide some demographic information and then the experiment started. Participants were shown a table with the original input data accompanied by a generated text from the NLG system trained on data–text pairings from one of the three semi-supervised learning approaches (no extension, data augmentation, pseudo-labeling). A selection was made of inputs that contained between 2 and 6 data elements, to keep the input data relatively understandable for participants. The generated texts were one sentence long (for CACAPO and WebNLG), or a few sentences long (between 1 and 6; for E2E). After viewing the input data and output texts, participants were asked to rate the texts on multiple items. Definitions of each item could also be found by hovering over the item.

We measured fluency using four seven-point Likert scale items based on Sundar (1999) and Clerwall (2014) (consistency was high, with α = .97). The items were introduced by “This sentence/short text is...”, followed by “Clear” (The overall message of the sentence/short text is clear.), “Coherent” (It is easy to follow the connections in the sentence/short text. The different pieces of information are connected in a correct way.), “Understandable” (The sentence/short text is written in a way that is easy to understand. There are no strange word choices or phrases that make the sentence/short text confusing.), and “Well-written” (The sentence/short text is fluent and easy to read.).

Correctness was measured using three seven-point Likert scale items based on Hoorn and van Wijngaarden (2010) (consistency was high, with α = .92). The items were introduced by “Based on the data table, the information in this sentence/short text is...”, followed by “Factual” (The sentence/short text only describes the data in the data table. There is no extra information being described in the sentence/short text that is not represented in the data table.), “Accurate” (The information in the data table is represented correctly in the sentence/short text. There are no mistakes in the names and numbers, for instance.), “Complete (All the (important) information from the data table is represented in the sentence/short text. There is no information missing in the sentence/short text that is represented in the data table.)”.

Grammaticality was measured using one multiple-choice question containing 4 options, based on Ross (1979). The question was introduced by: “How grammatically correct is this sentence/short text?” followed by (1) “The sentence/short text sounds perfect. I would use it without hesitation.”, (2) “The sentence/short text is less than perfect – something in it just doesn’t feel comfortable. Maybe lots of people could say it, but I never feel quite comfortable with it.”, (3) “Worse than 2, but not completely impossible. Maybe somebody might use the sentence/short text, but certainly not me. The sentence/short text is almost beyond hope.”, (4) “The sentence/short text is absolutely out. Impossible to understand, nobody would say it. Un-English.” For the results section, this domain was reverse-coded to make the scores better interpretable.

Finally, after rating all sentences/short texts, participants were fully briefed on the goal of the study, reminded of the contact addresses if they had more questions about the research, and thanked again for participation.

An error analysis was performed to get a better understanding of the exact errors that can be found in the NLG output, which in turn may help to improve the various systems. The 15 worst-scoring texts (on the average of all three measured constructs) in the quantitative human evaluation experiment for each system-dataset combination (a total of 180 texts) were analyzed by 7 human annotators, all experts in language and communication and proficient in both Dutch and English; none having previously seen the output of the various systems.

All annotators coded 19 sentences jointly and 23 sentences individually. Cohen’s Kappa for multiple raters (Davies and Fleiss 1982) was calculated for the jointly annotated part,⁸ resulting in κ = .45. This indicates moderate agreement (Landis and Koch 1977). Eleven error categories for these 180 texts were developed based on Castro Ferreira et al. (2019), and less straightforward error categories were accompanied by a short description with examples. See Table 6 for an overview of the items. If annotators selected “Other”, they were able to input text to describe the error category they felt they encountered.

The automatic analysis results are summarized in Table 7 for the overall datasets, and presented per domain in Table B.1. Additionally, Figure 5 shows the effects of increasing the synthetic data from the semi-supervised learning methods on BLEU scores. Inspection of Table 7 reveals a clear pattern: For CACAPO the pseudo-labeling approach consistently leads to the highest automatic metric scores, while for E2E and WebNLG data extension does not lead to better automatic scores overall. This pattern is consistent among all metrics, but we will zoom in on the BLEU differences in this section.

The results of the automatic metrics suggest that clear differences between datasets exist in the output quality achieved with the different semi-supervised learning approaches. For the English CACAPO dataset, BLEU scores, for instance, improved by 5.71 on average with the pseudo-labeling approach (30.50 to 36.21), but the data augmentation approach led to an average BLEU decrease of 6.13 compared with no extension (30.50 to 24.37). The positive effect on automatic metric scores was more noticeable for the Dutch part of the CACAPO dataset, where BLEU scores increased by 20.31 on average for the pseudo-labeling approach (33.94 to 54.25) and a 4.36 BLEU improvement for the data augmentation approach compared with the no extension approach (33.94 to 38.30).

The pseudo-labeling approach for WebNLG generally led to a decrease in automatic metric scores (compared with no extension) albeit relatively small, with an average BLEU decrease of 3.36. However, the BLEU decrease for the data augmentation approach (compared with no extension) was more noticeable, with a 20.20 BLEU decrease. Decreases in automatic metric scores for the semi-supervised learning approaches compared with no extension were also observed for E2E: A 15.54 BLEU score difference for pseudo-labeling and a 37.64 decrease for data augmentation (see Table 7).

Moreover, Figure 5 shows that an increase in synthetic training data leads to a decrease in BLEU scores for both E2E and WebNLG. This decrease could suggest a cascading of errors where issues in the synthetic data negatively impact the quality of the generated output. Alternatively, it could be that the texts in E2E and most WebNLG domains are relatively homogeneous. Introducing more deviations from these texts increases the diversity of the output, which results in lower scores on automatic metrics that use a golden standard. This could also explain the relatively low scores of the data augmentation approach, where the quantity of the deviations may have the biggest impact on the heterogeneity of the output. This possibility is further examined with the diversity metric scores, the quantitative and qualitative human evaluation, and the error analysis.

For Dutch CACAPO, the most extreme improvement was reached for the Weather dataset (a 52.38 improvement for pseudo-labeling compared to no extension; see Table B.1). It is possible that the small size and limited vocabulary of the original Weather training set was insufficient for a neural NLG system to be properly trained on, whereas it was when the extended training set from the semi-supervised learning approaches were applied. Figure 5 also shows that increasing the amount of data leads to higher BLEU scores for both Dutch and English CACAPO—that is, for the pseudo-labeling method. This provides further support for the notion that extending the training set has a positive impact on output quality.

Interestingly, for WebNLG, the pseudo-labeling approach performed well for the City and Monument domains where F1 scores in the pseudo-labeling step were noticeably worse (see Table B.1). This supports the notion that semi-supervised learning is mostly effective in situations where the original training set is small (as stated in H3), filling in the training data deficit by providing more examples.

An extension to the original study’s design was implemented where the configuration of T5 was initialized without the weights associated with the model for the training phase. The inclusion of this model provides a better overview of the baseline’s strength (i.e., the no extension model). Furthermore, a model was trained that leveraged training data enriched by both the pseudo-labeling as well as the data augmentation approach, to investigate whether a combination of the two impacts output quality.

Table 7 shows that for all datasets, a model initialized without weights performs markedly worse on all metrics compared with the models that do include the pretrained model weights. This is especially noticable for the English CACAPO dataset, where the model without weights clearly struggles. This corroborates findings by van der Lee et al. (2020), who also found that a deep learning model without pretrained data struggled with the CACAPO dataset. The characteristics of this dataset with its lack of repetition and variability make it difficult for such approaches to succeed.

Figure E.1 and Table 7 illustrate the effects of combining both data augmentation and pseudo-labeling. Generally, it performs similar to the best-performing semi-supervised learning approach, albeit a bit worse. Only for the English CACAPO dataset does the combination of the two approaches achieve higher scores, although the scores are almost similar to those achieved with the pseudo-labeling method. Combining the two approaches therefore does not seem to benefit output quality.

In any case, while these automatic metrics do not provide all-around support for the notion that semi-supervised learning combined with a language model increases output quality compared to an NLG system with a language model that is only finetuned on the original dataset (H1), the results are in line with H3: The beneficial effect of semi-supervised learning in the text quality metrics is only noticeable for the dataset (categories) that are small-scale or unbalanced. They are also in line with H5: The highest increase of the semi-supervised learning methods was observed for the Dutch CACAPO dataset.

A summary of the diversity metrics can be found in Table 8 on a dataset level, and on domain level in Table B.2. Furthermore, Figure 6 shows the impact of an increase in the synthetic data from the semi-supervised learning methods on coverage scores. Overall, data extensions generally lead to higher diversity scores. The semi-supervised learning approaches seem to generate more diverse output compared with no training set extension, but the semi-supervised learning approach that gives the most diverse output seems to differ per dataset (domain) and per diversity metric. Therefore, we will discuss the outcomes grouped by different diversity metrics.

Average sentence length and standard deviation of the sentence can be an indicator of perceived diversity in a text: Longer sentences tend to contain more variation, and bigger differences between sentence length make the output more heterogeneous. It could be expected that the pseudo-labeling approach in particular affects sentence length standard deviation if the sentences that this approach introduces are also of a varied sentence length, while the average sentence length should not change too much for CACAPO and E2E, as the newly introduced sentences come from similar sources as the sentences in the original training set. For WebNLG, changes in average sentence length can be expected for the pseudo-labeling approach as the sentences are from a different source (Wikipedia vs. Crowdsourced). The data augmentation approach is expected to keep the standard deviation of sentence length, and average sentence length similar, as this approach perturbs words but generally keeps sentence structure and length the same.

The effect of pseudo-labeling, however, is only partially according to expectations; pseudo-labeling obtained the highest standard deviation score for 9 of the 19 domains, but only for 1 dataset overall (see Table 8; Table B.2). In terms of sentence length, it only shows a clear difference (decrease) compared with the no extension approach for E2E, and only marginal differences for the other datasets and dataset domains. Data augmentation indeed shows similar scores for average sentence length and standard deviation of sentence length consistently among datasets and dataset domains.

The number of types, type-token ratios, percentage of novel texts, and the novelty score are all direct indicators of lexical diversity. For these metrics, the improvements of the semi-supervised learning approaches are also the most pronounced. The data augmentation and pseudo-labeling approach each seemed to perform best in terms of increasing lexical diversity on roughly half of the datasets and dataset domains.

For the English CACAPO, and E2E, data augmentation seemed to result in the highest lexical diversity scores, while these scores were highest for the pseudo-labeling approach in the case of Dutch CACAPO and WebNLG. For the Dutch CACAPO dataset, this may have to do with the nature of the language model used for data augmentation: BERT-large for English (Devlin et al. 2019) is likely better able to inject diverse perturbations compared with BERTje (de Vries et al. 2019), which is a Dutch translation of BERT-base.

For WebNLG it might have to do with the nature of the texts that were used for pseudo-labeling: The Wikipedia texts are probably more dissimilar (thus injecting more diversity) to the texts in the training set, compared with the pseudo-labeled texts used for the other datasets. It is worth noting that the novelty scores of the no extension approach for the WebNLG and E2E datasets are consistently close to 0, meaning that (almost) no new words have been introduced for these datasets that were not found in the training or development data. This absence of new words possibly showcases catastrophic forgetting (Greco et al. 2019) (which may occur when a training set is decently sized for finetuning), which leads to a neural NLG system forgetting about the language model’s capabilities.

Also worth noting is that the percentage of novel texts shows that (almost) all generated texts in CACAPO and E2E are different from the texts in the training data, thus creating a ceiling effect for these datasets. This has to do with the setup of the test set, where input data rarely, if ever, overlaps with the input data in the training and development sets. This metric is more interesting for WebNLG, where we see that non-novel sentences are only rarely generated with the data augmentation approach, while this does occur more frequently with the pseudo-labeling approach and no extension approach.

Coverage and local recall measure text retention in the output compared to the training set. Coverage is a global recall metric, measuring how many of the word types from the training and development sets were retained in the output. A higher coverage score suggests that more of the naturally occurring variation in the training and development set is retained in the test set. Local recall compares the content words in the generated output to those in the reference in the test set. A higher score on lexical variation metrics, paired with a high score on local recall, indicates that higher diversity was not obtained at the cost of meaningful content words.

Regarding coverage, we again see that the semi-supervised learning approaches consistently outperform the system that only used the standard training set. The highest coverage scores for WebNLG and Dutch CACAPO were achieved with pseudo-labeling, while data augmentation achieved the highest coverage score for English CACAPO and for E2E. Furthermore, Figure 6 shows that coverage scores are generally increased when the training set size is increased with synthetic data. This effect is most pronounced for pseudo-labeling, whereas an increase in training set size does not necessarily lead to higher coverage scores with data augmentation; while training size increases with data augmentation do seem to have a positive effect on coverage for CACAPO, they seem to decrease coverage scores for WebNLG and E2E.

For local recall, we see that the pseudo-labeling approach achieved the highest scores for Dutch and English CACAPO. The small-scale nature of these domains might stand to benefit more from the extra training data that the pseudo-labeling approach offers to retain commonly occurring content words. For WebNLG and E2E we see that the system trained on the original dataset retains the most content words, while the pseudo-labeling approach generally stays relatively close in terms of local recall scores. The data augmentation approach shows a more sizable drop-off, which makes sense as the content words were augmented for this semi-supervised learning approach.

Diversity metrics were also applied to the extension to the original study’s design where a T5 model’s configuration was initialized without the weights associated with the model and the T5 model that was enriched with both data augmentation as well as pseudo-labeling. These extensions assess the strength of the no extension baseline model, and whether combining the two semi-supervised learning approaches helps to fortify their individual strengths with regard to output quality and diversity.

Table 8 indicates that for all datasets, a model initialized without weights performs worse on most diversity metrics compared with the models that include the pretrained model weights. The model does achieve generally higher scores on average sentence length (for the CACAPO datasets) and standard deviation of sentence length (for all corpora), but paired with the lower number of types, coverage, novelty, and local recall scores, this seems to be an indicator of natural language degeneration (Holtzman et al. 2020). However, for the E2E dataset, this dropoff compared to the other models does not seem as sizeable compared with the other datasets. A reason for this may be that the lack of variety and repetition in this dataset, together with its size, allows for ample opportunity for the model to learn the appropriate model weights.

The effects of joint data augmentation and pseudo-labeling can be found in Figure E.2 and Table 8. Similar to the results of the automated metrics, it seems that this combination performs similar to the best-performing semi-supervised learning approach, albeit a bit worse. For the E2E and WebNLG dataset, this approach does seem to output more types, albeit only a small amount more. For other metrics, the combination of the two approaches achieves higher scores, although the scores are almost similar to those achieved with the pseudo-labeling method. Combining the two approaches therefore does not seem to have a marked impact on output diversity.

In support of H2, we find that semi-supervised learning increases the output diversity. Data augmentation seems to be the best method to achieve this for English CACAPO and E2E, whereas the pseudo-labeling approach achieves better diversity scores for Dutch CACAPO and WebNLG. The results also indicate that the neural NLG system without an semi-supervised learning approach suffers from catastrophic forgetting: Overfitting on the training set too tightly—which means a relative lack of output diversity as a result. We do not find a marked difference between datasets in terms of the effects of semi-supervised learning in general on output diversity. All datasets seem to benefit relatively equally from semi-supervised learning. Therefore, no clear effect of dataset type (crowdsourcing based vs. naturally occurring) and language (Dutch vs. English) was detected, which is contrary to H4 and H5, respectively.

In addition to the automatic evaluations, we also conducted a human evaluation to investigate perceived quality, by measuring fluency, correctness, and grammaticality. The overall scores are summarized in Table 9, and reported per sub-corpus in Table B.3. Inspection of Table 9 reveals a clear pattern: In almost all cases the highest scores are obtained with the pseudo-labeling approach.

This was further examined by conducting a series of linear mixed models, with semi-supervised methods and the datasets/dataset domains as independent variables and fluency, correctness, and grammaticality as the dependent variables. Linear mixed models enable us to control for the systematic variation due to the nested/clustered structure of the data caused by participants having rated multiple texts. Furthermore, linear mixed models are able to deal appropriately with unequal sample sizes (i.e., language and dataset in this experiment). To do so, we added dataset, domain, and participant as a nested random factor in the models.

Following van Miltenburg et al. (2019), we used the lme4 (Bates et al. 2015) package in R to build our linear mixed models using the lmer function and estimate p-values for the models, respectively. Datasets and dataset domains were investigated using separate models, and a separate model for each dependent variable was also necessary, meaning that a total of 6 models were created. Furthermore, “no extension” served as the reference level intercept for semi-supervised learning approach, and “E2E” as the reference level intercept for dataset and dataset domain. The emmeans package was used for pairwise comparisons when analyzing the differences between semi-supervised learning approaches, datasets, dataset domains, and interaction effects if the linear mixed model showed significant main effects or interaction effects. All models converged and the visual check of model assumptions using the residual plots indicated no signs of violations.

The model for fluency was significant (conditional R² = 0.23, marginal R² = 0.07, p < .001). The model showed a main effect of training type, but not of dataset, and also did not suggest any interaction effects. Table 9 represents the mean scores and training type differences, while the mixed model results are summarized in Table C.1.

Significant effects were further investigated using estimated marginal means with Bonferroni correction, which corroborated these findings. Only the pairwise comparisons for semi-supervised learning approaches showed significance: There was no significant difference in fluency between no extension (M = 4.73, SD = 2.05) and data augmentation (M = 4.74, SD = 2.06), but the fluency scores for the pseudo-labeling approach (M = 5.79, SD = 1.62) were significantly higher than both the data augmentation approach and no extension. This effect was robust across all dataset domains as well, with all but 4 domains showing the same pattern between semi-supervised learning approaches (see Table B.3).

Figure 7 illustrates these results: All datasets show a similar pattern, where the fluency scores between no extension and data augmentation are virtually similar, and the perceived fluency increases for pseudo-labeling. Sentences were thus perceived as more fluent when a training set was enriched with data created via the pseudo-labeling approach. Furthermore, perceived fluency scores were relatively similar for all the investigated datasets, and the fluency differences between the semi-supervised learning approaches were similar across all datasets.

Partly in support of H1, semi-supervised learning seems to increase the output quality compared with a language model only trained on the base training set. However, this is only the case for the pseudo-labeling approach and not for the data augmentation approach (RQ1). Furthermore, these fluency results do not support H3, nor H5: The beneficial effect of semi-supervised learning is consistent regardless of whether the original dataset is small-scale or large-scale (H3), Dutch or English (H5).

For correctness, the model was significant as well (conditional R² = 0.29, marginal R² = 0.11, p < .001). The model showed a main effect of semi-supervised learning approach, dataset, and an interaction between the two. Table 9 shows the mean scores and semi-supervised learning approach differences, and Table C.1 a summary of the mixed models for correctness.

The significant effects were further explored using estimated marginal means with Bonferroni correction. For semi-supervised learning type, there was no significant difference between no extension (M = 4.46, SD = 2.12) and the data augmentation approach (M = 4.47, SD = 2.12), but there were significantly higher correctness scores for the pseudo-labeling approach (M = 5.56, SD = 1.78) compared with both other approaches.

For dataset, there was no significant difference between E2E (M = 4.10, SD = 1.80) and WebNLG (M = 4.54, SD = 2.23), and no difference between English CACAPO (M = 5.14, SD = 1.84) and Dutch CACAPO (M = 5.35, SD = 1.81). However, both CACAPO datasets scored significantly higher on perceived correctness than both WebNLG and E2E. The main effects and interaction effect can be explained in more detail using Figure 8.

Regarding differences between datasets, differentiated per semi-supervised learning approach, we see that for no extension, WebNLG (M = 3.99, SD = 2.21) obtained significantly worse correctness scores compared with both Dutch CACAPO (M = 5.15, SD = 1.90) and English CACAPO (M = 4.93, SD = 1.86).

This pattern is similar when comparing the datasets enriched with data augmentation: WebNLG (M = 4.01, SD = 2.22) performed significantly worse in terms of correctness compared with both Dutch CACAPO (M = 5.15, SD = 1.88) and English CACAPO (M = 4.92, SD = 1.89). This consistency also makes sense when comparing the datasets’ scores on no extension to data augmentation: None of the datasets’ scores on data augmentation differ significantly from the scores they obtained on no extension.

The pattern is different for pseudo-labeling: The correctness scores for English CACAPO (M = 5.57, SD = 1.70), Dutch CACAPO (M = 5.75, SD = 1.58), and WebNLG (M = 5.63, SD = 1.82) are significantly higher for pseudo-labeling than they are for both no extension and data augmentation, resulting in no significant differences in correctness scores between the three datasets for pseudo-labeling.

However, the correctness score for E2E on pseudo-labeling (M = 3.86, SD = 1.71) is not significantly different compared to the correctness score E2E obtained on no extension and data augmentation. This results in E2E performing significantly worse on pseudo-labeling compared to English CACAPO, Dutch CACAPO, and WebNLG.

On a dataset domain level, the Astronaut, Food, and Monument domains of WebNLG in particular performed worse for WebNLG when the base training set or data augmentation was used. Furthermore, we see that the differences between semi-supervised learning approaches are most pronounced for the WebNLG domains, while the differences are not as clear for the English CACAPO and Dutch CACAPO domains (see Table B.3).

The data shows that overall, no extension and data augmentation performed similarly on perceived correctness across all datasets, and that pseudo-labeling also leads to the highest correctness scores as it does for fluency scores. However, this effect is moderated by the dataset. E2E did not achieve higher correctness scores with the pseudo-labeling approach compared to the other approaches, while the other datasets did.

Similar to the fluency results, the correctness scores are partially in support of H1, showing that the semi-supervised learning shows higher correctness scores compared with a language model only trained on the base training set, but only in the case of the pseudo-labeling approach and not for the data augmentation approach (RQ1). The correctness scores are partially in support of H3, as the beneficial effect of semi-supervised learning could not be found for the largest-scale dataset (E2E), but it could be found for WebNLG. Finally, the correctness scores do not seem to differ per language, which does not support H5.

The model for grammaticality was significant (conditional R² = 0.20, marginal R² = 0.07, p < .001). Similar to the fluency model, the grammaticality model showed a main effect of training type, but not of dataset, and also did not suggest any interaction effects. Mean scores for grammaticality and differences for training type are shown in Table 9 and the mixed model results are summarized in Table C.1.

The estimated marginal means with Bonferroni correction shows that no extension (M = 2.90, SD = 1.02) was not significantly different compared with data augmentation (M = 2.89, SD = 1.03) in terms of grammaticality, and that both no extension and data augmentation obtained significantly worse grammaticality scores compared with pseudo-labeling (M = 3.42, SD = 0.80).

Figure 9 shows a similar pattern as was found for fluency. The pseudo-labeling approach led to higher grammaticality scores than no extension and data augmentation did. Furthermore, the effects of grammaticality seem to be consistent among datasets: The datasets did not differ significantly from each other in terms of the obtained grammaticality scores, and performed similarly with the semi-supervised learning approaches.

The differences between semi-supervised learning approaches are also fairly stable on a category level, with 14 out of 19 domains having no significant difference between no extension and data augmentation, and pseudo-labeling performing significantly better than the other two approaches.

Similar to fluency, the results for grammaticality are partly in support of H1, as semi-supervised learning increased grammaticality scores compared to a language model only trained on the base training set, but only for the the pseudo-labeling approach (RQ1). The grammaticality results also do not support H3/H5 as dataset size (H3) and language (H5) did not alter the found effect of semi-supervised learning.

The goal of the error analysis was to investigate whether the different systems were prone to other kinds of errors, which could indicate points of attention for future work. Hence, a chi-square test was conducted with error type (see Section 4.3), semi-supervised learning approach (no extension, data augmentation, and pseudo-labeling), and dataset (Dutch CACAPO, English CACAPO, WebNLG, and E2E) as categorical variables. Pairwise comparisons per category between semi-supervised learning types were made using Bonferroni-adjusted z-tests for column proportions. Pairwise comparisons between the total scores per dataset were done using binomial tests with Bonferroni-adjusted p-values. Results of the comparisons are shown in Table 10.

The chi-square test did not show a significant difference in error proportions between the semi-supervised learning approaches for Dutch CACAPO (χ²(18) = 6.45, p = .994), English CACAPO (χ²(20) = 17.82, p = .599), WebNLG (χ²(20) = 21.11, p = .391), or E2E (χ²(18) = 28.52, p = .055), as well as all the datasets combined (χ²(20) = 26.43, p = .152).

However, the binomial tests with Bonferroni-adjusted p-values showed that the proportion of total errors for the pseudo-labeling approach, across all datasets, was significantly lower than the expected (33%, p < .001). No difference in proportion of errors was found between no extension and data augmentation. The pseudo-labeling approach yielded only around half of the errors of the other two approaches in total, and this effect was reasonably consistent for every dataset (see Table 10).

This indicates that differences found in the quantitative human analysis are generally not the result of distinct categorical error differences, but rather an overall difference in all errors combined.

Despite no overall effect being found, the pairwise comparisons do show a few significant differences. For WebNLG, a larger proportion of Other errors were found for the pseudo-labeling approach (15.4%) compared with the data augmentation approach (0.0%), but this difference is likely explained by the small number of error observations for this particular error category and dataset.

More interesting is the difference found for the E2E dataset regarding the number of missing information errors obtained by the data augmentation approach (8.1%), compared with the pseudo-labeling approach (37%), which could be the reason why the correctness score of the pseudo-labeling approach was lower for the E2E dataset compared with the other datasets in the quantitative human evaluation study.

Furthermore, the errors of all datasets combined showed that the pseudo-labeling approach had much fewer issues with the repetition of words/phrases (1.1% of errors for pseudo-labeling, compared with 11.7% and 13.0% for no extension and data augmentation, respectively). These issues are seen in sentences such as “Batchoy is eaten in the their, a the of the their spoken is is.” and may indicate an issue of underfitting, which is solvable by extending the training set via the pseudo-labeling approach. These findings are further elaborated upon with the qualitative evaluation.

In addition to the error analysis, the output of the system was further analyzed by focusing on the 15 lowest and 15 highest rated outputs per dataset × semi-supervised learning approach according to the human evaluation task. The bottom/top 5 of these outputs for each dataset × semi-supervised learning approach is displayed in Appendix D.

When looking at the highest rated texts, all texts obtained perfect or near-perfect ratings in the human evaluation task. Manual inspection also confirms that none of these sentences contain any clear errors: The texts are grammatical and capture the data (almost always) completely and comprehensively. Furthermore, on a general level it can be observed that the data–text pairs for no extension and data augmentation partially overlap, which further emphasizes the similar performance of both models.

Certain trends could also be deduced from these highest rated texts on a closer inspection. For the Dutch CACAPO dataset, output from the Weather and Accidents domain accounted for nearly 70% of the highest rated texts. This makes intuitive sense as texts from these domains in the training set are the most consistent in writing style. Furthermore, the data representations for these domains seem to be especially capable of capturing the most important information of the sentence. For instance, the data representation timePoint=”In_the_evening” weatherChange=”gradually” weatherType=”dry” becomes ”In the evening it gradually becomes dry”, which is quite close to the data representation.

This trend of a close link between the underlying data and the text leading to the highest scores is also visible for the other datasets. For the English CACAPO dataset, the stocks domain accounted for almost 60% of the highest rated texts. Especially prevalent is a template that contains the exchangeName, stockChange, stockPoints, and stockChangePercentage data types (e.g., exchangeName=”Dow” stockChange=”up” stockPoints=”288.38” stockChangePercentage=”1%”→ ”The Dow is up 288.38 points, or 1%.”). This template is likely to appear frequently in the training set with highly regular text representations, which in turn made it easier for the models to learn how to convert this data template into text.

For WebNLG, the models seem to find it easier to convert data representations with fewer triples into text. For no extension and data augmentation, there are no domains that clearly dominate the list of highest rated output. However, for pseudo-labeling, the top 6 are all outputs from the Building domain. WebNLG has data representations that stay relatively close to the domain’s topic (e.g., for the building domain this would be triples with predicates such as architect, floorCount, and location), and data representations that stray further away from the domain’s topic (e.g., for the building domain this would be triples with predicates such as demonym, or leaderName). The highest rated texts all seem to be the first type of data representations. This especially holds true for pseudo-labeling, which makes sense given that the enriched texts will likely be closer to the domain’s topic.

It is difficult to gauge which conditions make the models more effective at producing high quality output for E2E. At least for no extension and data augmentation, similar data representations can be found for the highest and lowest rated texts. For pseudo-labeling, most of the highest rated texts have a similar data representation, with name, eatType, food, area, familyFriendly, and near as the data types (e.g., name=”Giraffe”eatType=”restaurant”food=”French”area=”riverside”familyFriendly=”no”near=”Raja Indian Cuisine”. This is likely a frequent data representation template in the training set.

The lowest rated output in the human evaluation task clearly shows issues in the output. The most common issues seem to be in the production of grammatically correct texts, possibly due to underfitting. Hallucination and omission of data happen somewhat frequently, but making the connections between data points seems to have been the most difficult for the models. Here, overlap between no extension and data augmentation can also be found, in line with the similar performance found between the two systems in the human evaluation.

For the Dutch CACAPO dataset, almost 70% of the lowest rated texts are from the Weather and Sports domain. For the Weather domain, this has mostly to do with texts that verbalize wind information (e.g., ”Tonight the wind will become northeasterly in more and more places.”). For the Sports domain, it seems like the variety and lack of repetition in the training set has caused issues with producing grammatically correct texts. For most of these lowest rated texts, the issue is mostly with incorrect use of words, omission, or repetition of words (e.g., ”The man was tripped by an accident.”, ”The south to southwestern is moderate.”). In other cases, mostly the texts related to wind direction, the output is not necessarily incorrect, but may be uncommon information for participants to read.

For the English CACAPO dataset, two thirds of the lowest rated texts are from the Accidents and Stocks domain. The main issue for this dataset also does not seem to be hallucination or omission of data, but rather issues with the verbalized texts, possibly caused by underfitting (e.g., ”CLEVELAND, Ohio – A November 2014 quintuple homicide in Cleveland left one dead and four injured injured injured.”). The pseudo labeling approach suffers less from underfitting, but the worst rated texts may be caused by table information in the original articles that have been interpreted as text. For instance ”9pm: -8” is not a full sentence, but would provide enough information in a table.

Almost half of the lowest rated texts in the WebNLG dataset derive from the Food domain, although it is difficult to say why this domain is the most challenging for the models. In all cases, it seems that repetition and incorrect word use are the main issue that cause the low ratings (e.g., ”Amatriciana sauce is a traditional traditional from the Lazio region.”, ”The league is the Greeks of which the team played in the in.”). For pseudo-labeling, the previously mentioned distinction between close and further removed data representations seems to be the source of difficulties. In most cases for pseudo-labeling, the lowest rated sentences have data representations that are not close to the domain’s main topic.

For the lowest rated texts in the E2E dataset, it is also difficult to pinpoint the exact conditions that cause the models to struggle as the data representations seem similar to those of the highest rated texts. However, for pseudo-labeling, it seems that the high frequency of data types in the (enriched) dataset that are not frequent in the test set cause issues. The model hallucinates food types that are not represented in the data (e.g., ”The Cricketers is a restaurant that serves Samoan food.”). These food types (e.g., Samoan, Burmese, Saint Helenian) are only found in the enriched data for pseudo-labeling, where they occur regularly, which explains why only for E2E, pseudo-labeling had no positive effect on the correctness ratings in the human evaluation study.

This study investigated the potentially beneficial effect of semi-supervised learning in combination with a language model. More specifically, it investigated whether enriching a training set via the data augmentation approach (i.e., generating several variants of a training text by replacing certain words with synonyms and semantically similar words) and a pseudo-labeling approach (i.e., labeling unlabeled texts using an information extraction model trained on the existing labeled training data) could increase the performance of data-to-text NLG that already utilizes a large-scale language model (T5-large/mT5-large).

Previous work has found that semi-supervised learning could increase the output quality of an NLG system (e.g., Qader, Portet, and Labbé 2019; Schmitt et al. 2020; Su, Huang, and Chen 2020; Tseng et al. 2020; Chang, Demberg, and Marin 2021) and that utilizing language models for data-to-text NLG could help improve output quality as well (e.g., Kale and Rastogi 2020). However, not much is known about the combination of language models and semi-supervised learning in a data-to-text generation setting.

Therefore, it is not known whether semi-supervised approaches are still effective when language models are also used, and, if they are, under what conditions they are effective. Besides the type of semi-supervised learning as a condition, this study also investigated multiple datasets with different characteristics to see whether they affected the effectiveness of semi-supervised learning in combination with language models.

Partial support was found for the hypothesis that semi-supervised learning positively affects output quality (H1). When observing the results of the quantitative human evaluation, the pseudo-labeling approach consistently obtained higher scores on correctness, grammaticality, and fluency compared with the data-to-text system with a language model that was only trained on the base training set (hereafter: No extension approach).

This is in line with Sun et al. (2020), who found for a text classification task that a combination of a language model and pseudo-labeling led to the highest scores. It also corresponds with previous research that used joint learning systems for data-to-text generation (e.g., Qader, Portet, and Labbé 2019; Schmitt et al. 2020; Su, Huang, and Chen 2020; Tseng et al. 2020; Chang, Demberg, and Marin 2021), with the difference that this research compared it to a system where a language model was finetuned on the training set, rather than a system that was trained on merely the training set.

However, the data augmentation approach performed equivalently to the no extension approach in the quantitative human evaluation study and error analysis, and the output quality scores were generally worse than the quality scores yielded by the pseudo-labeling approach. These results suggest that the pseudo-labeling approach is a better semi-supervised learning approach than the data augmentation approach if the goal is to increase the output quality over a system that is only trained on the base training set (RQ1).

These results make intuitive sense: The data augmentation approach only makes subtle changes to the data that is represented in the text, and does not make any fundamental adjustments to the sentence structure compared with the original text, thus keeping the output relatively similar to that of the no extension approach. The pseudo-labeling approach introduces not only new sentence structures to the training set, but also a large amount of new data types and combinations of data. This increased variety in training data might lead to better generalizations and better handling of the diverse situations in the test set, subsequently leading to higher quality output and less underfitting issues (as was corroborated by the error analysis).

Thus, while previous research with different data augmentation approaches found data augmentation to increase performance compared with a system that was only trained on the training set (e.g., Kulhánek et al. 2021; Riabi et al. 2021; Tandon et al. 2018; Alberti et al. 2019; Chang et al. 2021; Kedzie and McKeown 2019), it may not result in text quality improvements if a language model is also implemented. It is possible that when the original training set is too small (which is common for NLG datasets), it may lead to an underfitted system. Underfitting can be reduced by implementing a language model that is finetuned on the training data, and by extending the training data with data augmentation. However, this combination of the two might be redundant for handling the underfitting issue.

The automatic metrics for output quality estimation suggest that the pseudo-labeling approach is the most effective approach for the CACAPO dataset—especially the Dutch CACAPO dataset—and that the no extension approach leads to the best output for E2E, and WebNLG (in concordance with H3 which states that smaller-scale datasets benefit more from semi-supervised learning, and H5 that states that Dutch datasets benefit more than English datasets).

However, the results of the quantitative human evaluation shows that the fluency and grammaticality outcomes are fairly consistent among datasets. The exception being correctness, where semi-supervised learning did not return higher ratings than the no extension approach for E2E. It is possible that this has more to do with the nature of the extensions used with E2E for the pseudo-labeling approach, as there was a clear correctness increase for the pseudo-labeling approach over the no extension approach for WebNLG.

The semantic parser that was applied on the extra texts for the pseudo-labeling approach seemed to be struggling (with F1 scores of only around 57% on the additional data), possibly due to overfitting on the original training set. These issues with the labels then eventually affected the correctness scores of the data-to-text system’s output (an example of cascading of errors [Castro Ferreira et al. 2019]). In turn, this would lead to worse connections between the data input and text output, as is also found by the error analysis.

Therefore, we believe our results are not in line with H3, nor H5 as the Dutch CACAPO corpus performed similarly to the other corpora, especially the English CACAPO corpus. These results are also not in line with previous research that found semi-supervised learning to especially bolster the results for small-scale datasets (Chang, Demberg, and Marin 2021). It might therefore be that the usage of language models nullifies the differences between datasets and their scale: Previous work has shown that language models enable few-shot and zero-shot learning (Bender et al. 2021), which could make the difference between dataset scale less important. The lack of difference between the Dutch and English datasets furthermore shows that, although the Dutch representation in mT5 is small (especially compared with the English representation in T5) it is still enough to overcome issues of underfitting.

The diversity differences between the various approaches were investigated using measures derived from van Miltenburg, Elliott, and Vossen (2018). We found support for H2: Semi-supervised learning generally seemed to increase output diversity. For every investigated dataset, one of the semi-supervised learning approaches scored the highest on almost each of the diversity metrics. This is also in line with previous findings by Kulhánek et al. (2021). This suggests that semi-supervised learning can (at least partially) solve the previously highlighted “catastrophic forgetting” problem (Greco et al. 2019), where a neural model is overfit too tightly during finetuning, which leads to the model forgetting about the diverse language in the language model.

The increase in diversity is relatively consistent among datasets (which does not support H4 that states that a larger increase in diversity is expected for crowdsourced datasets) and dataset language (which does not support H5). However, the semi-supervised learning approach that leads to the largest diversity increases differs per dataset. The pseudo-labeling approach is generally the most diversity-bolstering approach for CACAPO (nl) and WebNLG, whereas the data augmentation approach is the most effective approach for diversity increases for CACAPO (en) and E2E.

However, it is difficult to pinpoint which characteristics of the datasets lead to these differences. Diversity differences may be due to the richness in content words in the original dataset (which leads to more perturbations with the data augmentation approach), or lack thereof. Alternatively, it can be that the extra texts used for the pseudo-labeling approach contain more (or less) diverse language for some datasets, leading to more (or less) diverse verbalizations of the data-to-text output. Future research is necessary to investigate which characteristics of the dataset—and extra data introduced by the semi-supervised learning approaches—are most salient when the goal is to increase text diversity.

The current study found that semi-supervised learning is an effective technique for data-to-text generation, even when used in conjunction with a language model. The pseudo-labeling approach can increase both output diversity and quality, whereas the data augmentation approach is effective at increasing diversity, while keeping the quality consistent with a model only enriched with a language model.

Furthermore, this result seems to be consistent among datasets. It may be that a good language model nullifies the differences between the original datasets, meaning that, for instance, the scale of the dataset does not matter as much. This in turn would mean that enriching training data with a language model can already be helpful for non-English NLG tasks, for which it is generally more difficult to find large-scale datasets (Riabi et al. 2021). This would help to make neural NLG systems available for more people around the world without having to invest in large-scale datasets.

Two common semi-supervised learning approaches (pseudo-labeling and data augmentation) were explored in the design of the presented data-to-text system. However, for future research there are also other approaches to explore. For instance, data augmentation can be done in other ways, other than perturbing words with semantically similar words, as was done in this research, such as data augmentation in the form of self-training where new texts are generated from unlabeled data. Previous research has shown that such an approach to data augmentation is promising (e.g., Heidari et al. 2021; Jolly et al. 2022; Mehta et al. 2022).

Also of interest would be entity resolution/text matching (i.e., automatically connecting data with corresponding text), rather than pseudo-labeling (Ahmadi, Sand, and Papotti 2021). Many real-world companies have large collections of texts and related data, but are missing an explicit connection between the two (van der Lee et al. 2020). Such a task could help to produce large quantities of extra data, and may additionally help to tackle issues with mistakes in the original manual data annotations that could lead to bad texts.

Especially interesting would be the use of unlabeled texts as an in-domain language model, rather than the Transformers-based language models such as BERT and T5 that were trained on an immense variety of different domains. Sun et al. (2020) found that using such an in-domain language model made the use of BERT unnecessary, and that the combination of in-domain pretraining and pseudo-labeling resulted in the best model for a text classification task. The impact of training large Transformers models such as BERT and T5 on CO₂ emissions is well documented (Strubell, Ganesh, and McCallum 2019). If the use of unlabeled texts for (generally smaller-scale) in-domain pretraining and pseudo-labeling is indeed also effective for NLG, this might help to focus development on more efficient and eco-friendly development of single-domain language models, rather than unsustainably striving for larger and larger all-encompassing Transformers models.

The findings of the current study highlight the feasibility of automatically extending datasets in a way that neural data-to-text systems benefit from the extension, even if the original dataset is small in size. This can be beneficial for robot journalism, for instance, which still mostly relies on rule-based NLG. Newspaper organizations oftentimes have large archives of texts and data, but aligning the two manually on a scale that permits a neural model to be trained on it requires an amount of manpower that most newsrooms do not have available. By creating a small starting point and then using semi-supervised learning to create the full dataset, would be a less intensive undertaking—thus a smaller hurdle—for many companies to use neural data-to-text systems.