Soloist: Building Task Bots at Scale with Transfer Learning and Machine Teaching

Abstract We present a new method, Soloist,1 that uses transfer learning and machine teaching to build task bots at scale. We parameterize classical modular task-oriented dialog systems using a Transformer-based auto-regressive language model, which subsumes different dialog modules into a single neural model. We pre-train, on heterogeneous dialog corpora, a task-grounded response generation model, which can generate dialog responses grounded in user goals and real-world knowledge for task completion. The pre-trained model can be efficiently adapted to accomplish new tasks with a handful of task-specific dialogs via machine teaching, where training samples are generated by human teachers interacting with the system. Experiments show that (i)Soloist creates new state-of-the-art on well-studied task-oriented dialog benchmarks, including CamRest676 and MultiWOZ; (ii) in the few-shot fine-tuning settings, Soloist significantly outperforms existing methods; and (iii) the use of machine teaching substantially reduces the labeling cost of fine-tuning. The pre-trained models and codes are available at https://aka.ms/soloist.


Introduction
The increasing use of personal assistants and messaging applications has spurred interest in building task-oriented dialog systems (or task bots) that can communicate with users through natural language to accomplish a wide range of tasks, such as restaurant booking, weather query, flight booking, IT helpdesk (e.g., Adiwardana et al., 2020;Roller et al., 2020b;Gao et al., 2020;. The 1 TASK-ORIENTED DIALOG WITH A SINGLE PRE-TRAINED MODEL. In this paper, SOLOIST refers to both the proposed bot building method and the dialog model or system developed using the method. wide variety of tasks and domains has created the need for a flexible task-oriented dialog development platform that can support many different use cases while remaining straightforward for developers to use and maintain. A typical task-oriented dialog system uses a modular pipeline, which has four modules and executes sequentially (Young et al., 2013;Gao et al., 2019a), as shown in Figure 1(a). A natural language understanding (NLU) module identifies user intents and extracts associated information such as slots and their values from users' input. A dialog state tracker (DST) infers the belief state (or user goal) from dialog history. The belief state is often used to query a task-specific database (DB) to obtain the DB state, such as the number of entities that match the user goal. The dialog state and DB state are then passed to a dialog policy (POL) to select the next system action. A natural language generation (NLG) module converts the action to a natural language response.
Most popular commercial tools for dialog development employ the modular systems, including Google's Dialog Flow, 2 Microsoft's Power Virtual Agents (PVA), 3 Facebook's Wit.ai, 4 Amazon's Lex, 5 and IBM's Watson Assistant. 6 They are designed mainly to help develop systems manually, namely, writing code, crafting rules and templates. Unfortunately, even with these tools, building dialog systems remains a label-intensive, time-consuming task, requiring rich domain knowledge, reasonable coding skill, and expert experience. The cost of building dialog systems at scale (i.e., tens of thousands of bots for different tasks) can be prohibitively expensive. Figure 1: Illustration of a traditional modular task-oriented dialog system, an example for the model input, and the proposed model. The SOLOIST solution utilizes a single neural auto-regressive model in (c) to parameterize the sequential dialog pipeline in (a), with input sequence represented in (b). Different from GPT-2, the SOLOIST model learns to ground response generation in user goals and database/knowledge.
With the recent advances in neural approaches to conversational AI , researchers have been developing data-driven methods and neural models for either individual dialog modules or end-to-end systems. For example, recent attempts such as RASA (Bocklisch et al., 2017), ConvLab (Lee et al., 2019b;, and Conversation Learner (Shukla et al., 2020) are made to allow the use of data-driven approaches based on machine learning and machine teaching to develop dialog modules. End-to-end trainable dialog systems have also been studied (e.g., Wen et al., 2017;Zhao and Eskenazi, 2016;Lei et al., 2018;Gao et al., 2019a;Zhang et al., 2020b). Although these methods have achieved promising results, they require large amounts of task-specific labeled data for training, which are rarely available for new tasks in real-world applications.
In this paper, we propose a novel method of building task bots at scale, SOLOIST, which significantly eases the workflow of training and deploying dialog systems for new tasks, compared to existing tools and methods. Our approach is inspired by the recent success of applying transfer learning to natural language processing (NLP) tasks: Big language models pre-trained on large amounts of raw text (e.g., BERT , RoBERTa , and UniLM ) can be effectively fine-tuned for a wide range of NLP tasks with few in-domain labels. Recently, these pre-trained language models have also been employed to develop dialog modules such as NLU and DST Coope et al., 2020;. The proposed SOLOIST uses a similar pretraining-and-fine-tuning framework for building end-to-end dialog systems. We parameterize a task bot using a Transformer-based auto-regressive language model, which subsumes different dialog modules (i.e., NLU, DST, POL, and NLG) into a single neural model. Task bot building proceeds in two stages: (i) In the pre-training stage, initialized using GPT-2 , we train a Transformer-based, task-grounded, response generation model using large heterogeneous dialog corpora. The model learns the primary task completion skills such as DST and POL, and can generate dialog responses grounded in user goals and real-world knowledge for task completion.
(ii) In the fine-tuning stage, we adapt the pretrained SOLOIST model to complete a specific (new) task using a handful of task-specific dialogs via machine teaching, where training samples are generated by human teachers interacting with the system (Zhu, 2015;Shukla et al., 2020).
We show through a comprehensive empirical study that SOLOIST is an effective method of building task bots at scale by successfully transferring two capabilities from the pre-trained model to a new task bot: (i) the capability of NLU and NLG learned on raw text, and (ii) the capability of grounding system responses in user goals and real-world knowledge for task completion, learned on the out-domain dialog corpora.
SOLOIST achieves state-of-the-art performance on two well-studied task-oriented dialog benchmarks, lifting the combined score by 10 points in automatic evaluation, and the success rate by 20 points in human evaluation. In the few-shot fine-tuning settings, SOLOIST adapts to the new domain much more effectively than competing methods, achieving a reasonable success rate using less than 50 dialogs. The promising results demonstrate the potential of the new method for developing task bots at scale. Instead of collecting, labeling data, and building one bot per task, we can pre-train a task-grounded response generation model, and adapt it to new tasks via transfer learning and machine teaching.

An Auto-Regressive Model for Dialog
The modular dialog system in Figure 1 constitutes a data processing pipeline that produces a sequence, through concatenating the input-output pair of each module along the generation process. Each consecutive pair in this sequence plays the role of annotated data for the corresponding module. Ideally, when the entire sequence is available, the data generation process of a dialog system (NLU, DST, POL, NLG) can be formulated as a single auto-regressive model. GPT-2 (Radford et al., 2019) is a state-ofthe-art (SoTA) auto-regressive language model trained on large amounts of open Web text data. Although after being fine-tuned using conversational data, GPT-2 can respond to users with realistic and coherent continuations about any topic of their choosing (Zhang et al., 2020c), the generated responses are not useful for completing any specific task due to the lack of grounding. SOLOIST inherits GPT-2's capability of producing human-like responses. Nevertheless, unlike GPT-2, SOLOIST is pre-trained to generate responses grounded in user goals and real-world knowledge for task completion. While GPT-2 is a language model for text prediction, SOLOIST is a stateful decision-making model for task completion, with the capabilities of tracking dialog states, selecting best system actions, and so on. Thus, SOLOIST is pre-trained using task-oriented dialog sessions annotated with grounding information, i.e., user goals, dialog belief states, DB states, and system responses. Specifically, each dialog turn in our training data is represented as: where s is the dialog history up to the current dialog turn, b is the dialog belief state acquired from human annotation, c is the DB state automatically retrieved from a database using b, and r is the delexicalized dialog response, from which the system response in natural language can be generated using some automatic post-processing. Each item in x is by itself a sequence of tokens, as illustrated by the examples in Figure 1(b). Thus, it is natural to treat the concatenation of them as a long sequence for model training, as shown in Figure 1(c). We pre-train the SOLOIST model using publicly available heterogeneous dialog corpora with labels of belief states and DB states. The pre-trained model can be fine-tuned to any new task to generate responses grounded in task-specific user goals and a database.

Task-Grounded Pre-Training
Given training data of N samples D = {x n } N n=1 , our goal is to build a neural model parameterized by θ to characterize the sequence generation probability p θ (x). We use a multi-task objective for learning θ, where each task is a self-supervised learning task.
To leverage the sequential structure of a task-oriented dialog system, the joint probability p(x) can be factorized in the auto-regressive manner as: where the factorization from (2) to (3) is based on the fact that p(c|b, s) = p(c|b) = 1, because the DB state c is obtained using a deterministic database-lookup process given a belief state b (e.g., via an API call). Note that (3) decomposes the joint distribution modeling problem into two sub-problems: belief state prediction p(b|s) and grounded response generation p(r|c, b, s). Since b and r are sequences, we can further factorize them in the left-to-right auto-regressive manner, respectively.
Task 1: Belief Prediction. For a belief state sequence of length T b , we define the objective of predicting the belief state as: where b <t indicates all tokens before t.
Task 2: Grounded Response Generation. A delexicalized response of length T r , r = [r 1 , · · · , r T r ], is generated by our model token-by-token from left to right, grounded in dialog history c, belief state b and DB state s. The corresponding training objective is defined as We apply a binary classifier on top of the feature to predict whether the items in the sequence are matched (y = 1) or mismatched (y = 0). The contrastive object is cross-entropy defined as: We generate three types of negative samples x , each of which is chosen with probability 1/3: (i) negative belief, where only the belief state item is replaced (ii) negative response, where only the response item is replaced (iii) negative belief + response, where both the belief state and response items are replaced.
Full Pre-Training Objective. θ is learned via maximizing the log-likelihood over the training dataset D, using a joint objective that combines (4), (5) and (6): Figure 1(c) illustrates the model architecture and learning objectives. The model is auto-regressive in a left-to-right manner, with each of the three training tasks labeled on its corresponding output (i.e., sub-sequence separated by a special token).
Implementation Details. Each dialog turn in training data is processed to form a sequence of tokens consisting of four items (s, b, c, r). For example, the dialog turn of Figure 1  This sequence, tokenized using byte pair encodings (Sennrich et al., 2016), can be readily used for multi-task training, as shown in Figure 1(c). The implementation of SOLOIST is based on Huggingface PyTorch Transformer (Wolf et al., 2020). The task-grounded pre-training of SOLOIST uses the public 117M-parameter GPT-2 as initialization.

Fine-Tuning and Machine Teaching
When deploying SOLOIST to a new task, we collect task-specific x in the same format as that used for pre-training as (1). When x is available, the conventional fine-tuning procedure is utilized: we use the same multi-task objective of (7) to update θ to adapt the model to complete the new task using labeled task-specific dialogs.
In real applications, annotated task-specific data is often unavailable, or noisy/incomplete beforehand. One may deploy the dialog system and acquire high-quality task-specific labels (e.g., belief state and system response) for each dialog turn using machine teaching. Machine teaching is an active learning paradigm that focuses on leveraging the knowledge and expertise of domain experts as ''teachers''. This paradigm puts a strong emphasis on tools and techniques that enable teachers-particularly non-data scientists and non-machine-learning experts-to visualize data, find potential problems, and provide corrections or additional training inputs in order to improve the system's performance (Simard et al., 2017;Zhu, 2015;Williams and Liden, 2017;Shukla et al., 2020).
We proceed fine-tuning using Conversation Learner (Shukla et al., 2020), a machine teaching tool, in the following steps: (i) Dialog authors deploy the pre-trained SOLOIST model for a specific task. (ii) Users (or human subjects recruited for system fine-tuning) interact with the system and generate human-bot dialog logs. (iii) Dialog authors revise a dozen of training samples by selecting representative failed dialogs from the logs, correcting their belief and/or responses so that the system can complete these dialogs successfully, as illustrated in Figure 2. The corrected task-specific dialog turns are used to fine-tune the model. Implementation Details. To adapt a pre-trained SOLOIST to a new task in our experiments, we always fine-tune SOLOIST using a small amount of pre-collected task-specific dialogs, and then continue to fine-tune it via machine teaching, as detailed in Section 3.3. Training examples are truncated to ensure a maximal length of 512. The pre-trained models are fine-tuned with a mini-batch of 6 on 8 Nvidia V100 until no progress is observed on validation data or up to 10 epochs. Nucleus sampling  is used for decoding, where the sampling top-p ranges from 0.2 to 0.5 for all our models. The best setup of hyper-parameters is selected through grid-search on the validation set. For the machine teaching experiment,pre-trained models are fine-tuned with SGD on a single Nvidia V100.

Experiments
This section evaluates the proposed SOLOIST to answer three questions: Q1: How does SOLOIST perform on standard benchmarks compared to SoTA methods? Q2: Does SOLOIST meet the goal of effectively generalizing to new domains in the few-shot fine-tuning setting? Q3: how effective machine teaching is for fine-tuning? Note that we employ the conventional fine-tuning method without machine teaching for a fair comparison when studying Q1 and Q2.

Experimental Setup
Dialog Datasets for Fine-Tuning. We validate the end-to-end dialog system performance of SOLOIST on two well-studied datasets. (i) Cam-Rest676 (Wen et al., 2017) is a single-domain taskoriented dialog corpus. It contains 408/136/136 dialogs for training/validation/testing, respectively. Following Lei et al. (2018), we delexicalize each token that occurs in the ontology with its slot Figure 2: Illustration of the machine teaching process using conversion learner. The human-bot conversion log in (a) can be edited via correcting its belief state in (b), and selecting/inserting a more appropriate response in (c). names such as restaurant name, phone number, and postcode. (ii) MultiWOZ dataset ) is a multi-domain task-oriented dialog dataset. It contains 8438/1000/1000 for training/validation/testing, respectively. Each dialog session contains 1 to 3 domains, such as Attraction, Hotel, Hospital, Police, Restaurant, Train, and Taxi. MultiWOZ is inherently challenging due to its multi-domain setting and diverse language styles.
Automatic Evaluation Metrics. Following , Inform, Success, and BLEU scores are reported. The first two metrics relate to the dialogue task completion-whether the system has provided an appropriate entity (Inform) and then answered all the requested attributes (Success). BLEU evaluates how natural the generated responses are compared to that generated by human agents. A combined score (Combined) is also reported using Combined = (Inform + Success) × 0.5 + BLEU as an overall quality measure.
Baselines. We compare SOLOIST with several strong baselines, which hold SoTA on the Cam-Rest676 or MultiWOZ datasets. (i) Multi-Action Data Augmentation (DAMD) (Zhang et al., 2020b) is a modular system, where each dialog module is implemented using a neural network, and the whole system is trained in an end-to-end manner. (ii) Sequicity (Lei et al., 2018) is similar to DAMD except that it does not use multi-action data augmentation.
(iii) GPT fine-tuning (Budzianowski and Vulić, 2019) is fine-tuned on GPT-2 to generate re-sponses based on the dialog state and history. (iv) ARDM (Wu et al., 2019b) utilizes GPT-2 as the pre-trained model to learn to generate role-aware responses given dialog context. The model has to work with a separate dialog state tracker for task completion. (v) HDSA ) is a modular dialog system, which generates responses using a BERT-based dialog policy and graph structure dialog act representations.

End-to-End Evaluation
CamRest676. Table 2 shows the result and lists annotations used by different models. SOLOIST achieves the best scores in all the metrics. ARDM performs similarly to SOLOIST in terms of Success and BLEU. However, ARDM cannot track dialog states and requires a separately trained state tracker to accomplish tasks. GPT-2 fine-tuned with task-specific data works reasonably well but lags behind SOLOIST by a large margin. Sequicity, which uses a jointly trained model with belief state and policy annotations, underperforms SOLOIST. This result also shows that, compared to other end-to-end models, SOLOIST not only achieves better performance but requires lower labeling cost for fine-tuning due to the use of task-grounded pre-training. Table 3. SOLOIST achieves the best performance in terms of Inform, Success, and Combined, lifting the previous SoTA by a significant margin (e.g., about 10 points improvement in Combined over DAMD). SOLOIST also outperforms the method of Ham et al. (2020), where GPT-2 is fine-tuned and applied for end-to-end dialog modeling. Compared to the   classical modular dialog systems such as DAMD, SOLOIST uses a much simpler architecture and requires much lower labeling effort. For example, SOLOIST requires only the belief states, while DAMD requires additional annotations for task definition (i.e., defining the intents, slots, and the corresponding value ranges) and dialog acts.

Few-Shot Evaluation
It is desirable for task bots to effectively generalize to new tasks with few task-specific training samples. Thus, the few-shot fine-tuning setting is a more realistic setting for evaluating dialog systems. Unfortunately, the existing task-oriented dialog benchmarks typically contain for each task hundreds to thousands of dialogs. Therefore, we re-organize CamRest676 and MultiWOZ to simulate the few-shot fine-tuning setting for end-to-end evaluation. 7 We sample from the MultiWOZ dataset the dialog tasks that contain only one domain. Attraction, Train, Hotel, and Restaurant domains are used. We do not use the domains of Police, Taxi, and Hospital, as they do not require explicitly tracking dialog states for task completion. For each domain, we randomly sample 50 dialog sessions for training and validation and 200 dialog sessions for testing. The only exception is the  Table 5: End-to-end evaluation on CamRest676 in the few-shot fine-tuning setting.
Attraction domain, which has 100 sessions for testing. For CamRest676, we randomly sample 20 sessions. Details are shown in Table 4. Table 5 and 6 report the end-to-end performance in the few-shot fine-tuning settings on CamRest676 and MultiWOZ, respectively. On all the domains, SOLOIST obtains substantially better performance in all the metrics. Removing task-grounded pre-training significantly hurts the performance of SOLOIST, although SOLOIST    without task-grounded pre-training still consistently outperforms DAMD in all the domains. SOLOIST without task-grounded pre-training is conceptually similar to Ham et al. (2020), but is architecturally simpler and needs fewer annotations. The result verifies the importance of task-grounded pre-training on annotated dialog corpora, allowing SOLOIST to learn how to track dialog and database states to accomplish a task.
To study the impact of using larger model size, we build a large version of SOLOIST, SOLOIST L , which is task-grounded pre-trained on the same data but using GPT-2 medium with 345M parameters as initialization. SOLOIST L consistently outperforms SOLOIST by a large margin. It indicates that a larger model is a better few-shot learner, exhibiting stronger generalization ability with limited in-domain data. We leave it to future work to significantly scale up SOLOIST.
We conduct experiments to fine-tune SOLOIST by varying the percentage of task-specific training samples, ranging from 1% (80 examples) to 20% (1600 examples), on the MultiWOZ dataset. As shown in Table 7, SOLOIST consistently outperforms DAMD for a wide range of dataset sizes, and the improvement is more substantial when smaller numbers of in-domain examples are used for fine-tuning.

Machine Teaching Results
The machine teaching module of Conversational Learner (CL) (Shukla et al., 2020) allows human teachers (dialog authors) to select and visualize dialogs, find potential problems, and provide corrections or additional training samples to improve the bot's performance. We use CL to evaluate the effectiveness of machine teaching for task bot fine-tuning. In our experiment, we first sample 10 dialogs from each domain to fine-tune SOLOIST as described in Section 3.3. The result is presented in the first row of Table 8. We then deploy the model to interact with human users via CL. The row of SOLOIST+Teach shows the result of machine teaching, where a human teacher has manually corrected 5 dialogs, which are recommended by CL using a ranking heuristic based on perplexity. The corrections are utilized to continually fine-tune the deployed system. Table 8 shows that SOLOIST+Teach consistently improves Combined by a large margin compared with that without human teaching. SOLOIST+Extra is used as an ablation baseline, where 5 randomly selected dialogs with full annotations from experts are added as extra examples to fine-tune the model. It shows lower performance than machine teaching. Figure 3 demonstrates the performance of SOLOIST in Restaurant by repeating the above machine teaching process in multiple iterations. We observe that in the second iteration of machine teaching SOLOIST+Teach improves Combined by more than 8 points while SOLOIST+Extra achieves 5 points higher. The result demonstrates the effectiveness of our two-step fine-tuning scheme to deploy SOLOIST for a new task (domain). In terms of machine teaching cost, taking the restaurant domain as an example, we assume that one slot-value pair of belief state correction counts as one edit and a response correction counts as ten edits. The total numbers of edits for SOLOIST+Teach and SOLOIST+Extra are 61 and 396, respectively, suggesting that machine teaching reduces the labeling cost by 6×.

Component-Wise Evaluation
This section evaluates SOLOIST on two NLU tasks (i.e., intent classification and slot filling), the DST task and the response generation task. We show that although SOLOIST is an end-to-end dialog model, it also performs well on these component tasks.

Intent Classification
The task is to classify a user utterance into one of several pre-defined classes (intents). We follow the experiment setting of . The last hidden state of SOLOIST is used as the sequence representation for classification. Several baseline methods are used for comparison. BERT-fixed and BERT-tuned are fine-tuned on BERT, with BERT parameters fixed   .
and updated during fine-tuning, respectively. A linear classifier with a softmax layer is added on top of BERT for classification. Universal Sentence Encoder and ConveRT are sentence encoders tailored for modeling sentence pairs, and are trained for optimizing the conversational response selection task. The results in Table 9 show that SOLOIST is comparable with SoTA intent classification models. SOLOIST is the best performer when the full dataset is used for fine-tuning but its performance deteriorates more quickly than USE+ConveRT when fewer samples are used for fine-tuning. It is interesting to investigate whether incorporating intent classification tasks in task-grounded pre-training can boost SOLOIST's performance. We leave it to future work.
Slot Filling. We follow the experiment setting of Coope et al. (2020) and formulate slot filling as a turn-based span extraction problem. The results in Table 10 show that SOLOIST performs significantly better than the SoTA method Span-ConveRT, a variant of ConveRT designed explicitly for slot filling. The gap is wider when fewer examples are used for training. For example, when 64 samples are used for training, SOLOIST outperforms Span-ConveRT by 20 points in F1 score.
Dialog State Tracking. We compare the dialog state tracking capability of SOLOIST with several strong baselines on MultiWOZ 2.0 and 2.1. The results in Table 11 show that SOLOIST achieves the best performance on MultiWOZ2.1 and similar performance to DST-Picklist , which requires pre-defined task ontology to guide state tracking. In comparison with Simple-TOD (Hosseini-Asl et al., 2020) that is based on GPT-2,  SOLOIST obtains 1.13% higher joint goal accuracy. We attribute the gain to the task-grounded pre-training that equips SOLOIST with task completion skills including dialog state tracking.
Context-to-Response. In this task, systems need to generate responses given the ground-truth belief state and DB search result (Wen et al., 2017). The results on MultiWOZ 2.0 are shown in Table 12. SOLOIST achieves the best performance in terms of Inform and Success but performs slightly worse in BLEU. The Combined score of SOLOIST is comparable with the current SoTA method DAMD. However, DAMD uses the labels of dialog act on both the user and system sides, which demands significantly higher labeling efforts than SOLOIST for model training. HDSA achieves the best BLEU score. Compared with HDSA, SOLOIST is much simpler and able to perform better in terms of Combined. SOLOIST outperforms ARDM in Combined. It is worth mentioning that ARDM cannot perform dialog state tracking and requires an extra dialog state tracker to accomplish tasks. These results show that SOLOIST can learn dialog policies accurately and generate natural language responses in the multi-domain scenario.

Human Evaluation Results
We conduct human evaluation to assess the quality of SOLOIST interacting with human users. Following the evaluation protocol in the DSTC8 track 1 challenge (Kim et al., (Zhong et al., 2018) 35.57 -GCE (Nouri and Hosseini-Asl, 2018) 36.27 -FJST (Eric et al., 2020) 40.20 38.00 HyST (Goel et al., 2019) 44.24 -SUMBT (Lee et al., 2019a) 46.65 -TOD-BERT  -48.00 Neural Reading  41.10 -TRADE (Wu et al., 2019a) 48.62 45.60 COMER  48.79 -NADST  50.52 49.04 DSTQA (Zhou and Small, 2019) 51.44 51.17 SOM-DST  51.38 52.57 DST-Picklist  53.30 -MinTL  52.10 53.62 SST (Chen et al., 2020) 51.17 55.23 Tripy (Heck et al., 2020) -  to converse with SOLOIST to achieve the goal and judge the overall dialog experience at the end of a session using four metrics. (i) Success evaluates task completion. (ii) Under. (language understanding score) ranging from 1 (bad) to 5 (good) indicates the extent to which the system understands user inputs. (ii) Appr. (response appropriateness score) scaling from 1 (bad) to 5 (good) denotes whether the response is appropriate and human-like. (iv) Turns is the average number of turns in a dialog overall successful dialog sessions. Turks are further required to write down a justification of giving a specific rating. In total, 120 dialog sessions are gathered for analysis. Table 13 shows the human assessment results on MultiWOZ. The results are consistent with the automatic evaluation. SOLOIST achieves substantially better performance than other systems over all the metrics. Moreover, SOLOIST outperforms the DSTC8 Track 1 Winner by a much larger margin in Success (+20 points) in human evaluation than that in automatic evaluation (+10 points in Table 3). We attribute this to the fact that Turks use more diverse language to interact with the target bots in interactive human evaluation than that in the pre-collected MultiWOZ dataset and the use of heterogeneous dialog data for task-grounded pre-training makes SOLOIST a more robust task bot than the others. In many test cases against SOLOIST, Turks comment that they feel like they are talking to a real person.   Figure 4 depicts a dialog example where a user interacts with SOLOIST to complete a multi-domain task. The user starts the conversation by asking for a recommendation of a museum in the center of town. SOLOIST identifies the user intent, and provides a recommendation based on the search result from an attraction DB. Then, the user wants to book a table in a restaurant in the same area. We can see that through the conversation, SOLOIST develops belief state, which can be viewed as the system's understanding of what the user needs and what is available in the DB. Based on the belief state and DB state, SOLOIST picks the next action, either asking for clarification or providing the user with information being requested. This example also demonstrates that SOLOIST is able to deal with some NLU challenges displayed often in human conversations, such as co-reference resolution. For example, SOLOIST understands that the ''same area'' at Turn 5 refers to ''centre of town'', and then identifies a proper entity from the restaurant booking DB to make the reservation.

Related Work
Dialog Systems. Dialog systems are typically grouped into two categories, task-oriented sys- its variants that ground language generation in the prescribed control codes such as CTRL (Keskar et al., 2019) and Grover (Zellers et al., 2019), or latent variables such as Optimus .
Recently, pre-trained language models have been adopted to develop task-oriented and chitchat dialog systems. To name a few examples of chit-chat dialog systems: DialoGPT (Zhang et al., 2020c), TransferTransfo (Wolf et al., 2019) and CGRG (Wu et al., 2020b) adapt GPT-2 using human conversational data for response generation. Plato (Bao et al., 2020) pre-trains a discrete latent variable model for response generation. Meena (Adiwardana et al., 2020) and BST (Roller et al., 2020b) pre-train large models on conversational data and have demonstrated expressive performance in generating social chit-chat dialogs. For task-oriented dialogs, Mehri et al. (2019a) explores different pre-training methods for dialog context representation learning. TOD-BERT  adapts the pre-trained BERT to achieve strong performance on four dialog sub-tasks. ConveRT  pre-trains a model on Reddit data for intent classification and response selection. Span-ConveRT (Coope et al., 2020) extends the framework to entity extraction. SC-GPT  uses a pre-trained language model to convert a dialog act to a natural language response. All these works use the pre-training and fine-tuning framework. However, they follow the modular architecture of task bots, and the pre-trained models are used for improving individual dialog modules such as NLU and DST. SOLOIST generalizes these methods to the entire dialog pipeline, building an end-to-end dialog system. Although these methods have achieved promising results, they are designed for specific domains, rendering difficulties in generalizing to multidomains such as MultiWOZ. Dialog models that can handle multi-domain tasks are studied in (Pei et al., 2019;Budzianowski and Vulić, 2019;Mehri et al., 2019b;Zhao et al., 2019;Wu et al., 2019b;Zhang et al., 2020b;Peng et al., 2017). However, these works require large amounts of in-domain labels to achieve good performance. In contrast, SOLOIST can effectively adapt to a new task in the few-shot fine-tuning settings.
The most related work to ours is Ham et al. (2020), which is the first attempt to fine-tune GPT-2 to build end-to-end dialog models. Hosseini-Asl et al. (2020) take a similar approach, and is a concurrent work of SOLOIST. However, SOLOIST differs from these two methods in two major aspects. The first is the use of task-grounded pre-training that allows SOLOIST to learn primary task completion skills, such as tracking dialog states and select system actions. These skills can be easily reused and adapted (e.g., via few-shot fine-tuning) to solve new dialog tasks, leading to a much higher task success rate, as reported in Section 3. The second is that the annotation cost required for training SOLOIST is much lower than that of Ham et al. (2020) or Hosseini-Asl et al. 2020. Training SOLOIST requires only belief states as labels. But training of Ham et al. (2020) and Hosseini-Asl et al. (2020) requires labeling each dialog turn with dialog acts. In addition, while SOLOIST is end-to-end trainable, the other two models are not and need heuristic rules to handle different database search conditions.

Conclusion
SOLOIST is a method of building task bots at scale with transfer learning and machine teaching. Unlike GPT-2, SOLOIST is pre-trained in a task-grounded manner. So, it can generate responses grounded in user goals and real-world knowledge for task completion. Experiments show that SOLOIST creates new SoTA on two popular task-oriented dialog benchmarks, and that SOLOIST outperforms existing methods by a large margin in the few-shot fine-tuning settings where only a limited number of task labels are available for fine-tuning.
We hope that SOLOIST can inspire dialog researchers and developers to comprehensively explore the new paradigm for building task bots based on task-grounded pre-training and finetuning via machine teaching, and improving the recipe we present in this paper, namely, formulating task-oriented dialog as a single auto-regressive language model, pre-training a task-grounded response generation model on heterogeneous dialog corpora, and adapting the pre-trained model to new tasks through fine-tuning using a handful task-specific examples via machine teaching.