Knowledge-Grounded Dialogue Generation with Pre-trained Language Models

We study knowledge-grounded dialogue generation with pre-trained language models. To leverage the redundant external knowledge under capacity constraint, we propose equipping response generation defined by a pre-trained language model with a knowledge selection module, and an unsupervised approach to jointly optimizing knowledge selection and response generation with unlabeled dialogues. Empirical results on two benchmarks indicate that our model can significantly outperform state-of-the-art methods in both automatic evaluation and human judgment.


Introduction
With advances in neural machine learning (Sutskever et al., 2014;Gehring et al., 2017;Vaswani et al., 2017) and availability of the huge amount of human conversations on social media (Adiwardana et al., 2020), building an open domain dialogue system with data-driven approaches has attracted increasing attention from the community of artificial intelligence and natural language processing. In this work, we are interested in generative approaches. Generative models for open domain dialogues are notorious for replying with generic and bland responses, resulting in meaningless and boring conversations . Such deficiency is particularly severe when human participants attempt to dive into specific topics in conversation (Dinan et al., 2019). As a result, there is still a big gap between conversation with existing systems and conversation with humans.
Very recently, there emerge two lines of research that seem promising to bridge the gap. One is to apply large-scale pre-trained language models, such as GPT-2 (Radford et al., 2019) The Star Trek Canon of the series an animated had 5 spin offs. A I watched a little of the next generation but could not get into it like i did with the original show . Response Human These adventures went on but were short lived and six feature films. DialoGPT I think it's worth it.  (Dinan et al., 2019) . such as DialoGPT (Zhang et al., 2019c) have exhibited compelling performance on generating responses that make sense under conversation contexts and at the same time carry specific content for keeping the conversation going. While the giant language models can memorize enough patterns in language during pre-training, they only capture "average" semantics of the data (Zhang et al., 2019c). As a result, responses could still be bland or inappropriate when specific knowledge is required, as illustrated by the example in Table 1. The other line is to ground dialogue generation by extra knowledge such as unstructured documents (Zhao et al., 2020). By the means, the documents (e.g., wiki articles) serve as content sources, and make a dialogue system knowledgeable regarding to a variety of concepts in discussion. However, collecting enough dialogues that are naturally grounded on documents for model training is not trivial. Although some benchmarks built upon crowd-sourcing have been released by recent papers (Zhou et al., 2018b;Dinan et al., 2019;Gopalakrishnan et al., 2019), the small training size makes the generation models generalize badly on unseen topics (Dinan et al., 2019) and the cost of building such data also prevents from transferring the techniques proved on the benchmarks to new domains and new languages.
Encouraged by the results on pre-training for dialogue generation and knowledge-grounded dialogue generation, and motivated by the problems in both sides, we consider bringing the two together in this work. Specifically, we propose knowledgegrounded dialogue generation with pre-trained language models in order to endow a generative model with both rich knowledge and good generalization ability 1 . The challenge is that pre-trained language models often set constraints on the maximum number of tokens they can handle (e.g., the maximum number for GPT-2 (Radford et al., 2019) is 1024), and thus hinders exploitation of the knowledge text which could be rather long and redundant (e.g., in Wizard of Wikipedia (Dinan et al., 2019), on average each conversation context is associated with 61.2 sentences retrieved from wiki articles, and the average number of tokens in the extra knowledge is 1625.6). Indeed, the conflict between model capacity and the ability required for processing long knowledge input represents an essential obstacle for applying pre-trained language models to knowledge-grounded dialogue generation, since on the one hand we always have to set up an upper bound to the capacity of pre-trained models in order to handle massive text corpus, and on the other hand we need to keep sufficient candidates with rich enough content in the procedure of response generation in order to guarantee the recall of relevant knowledge.
To overcome the challenge, we consider equipping the pre-trained response generation model with a knowledge selection module whereby the redundant knowledge input is slimmed with relevant information (regarding to conversation contexts) kept to meet the capacity constraint. While some recent papers on knowledge-grounded dialogues have paid attention to the problem of knowledge selection (Lian et al., 2019;Kim et al., 2020;Ren et al., 2019), the knowledge selection module is either deeply coupled with the specially configured models (Lian et al., 2019;Ren et al., 2019) and thus is incompatible with the pre-trained language models, or it is learned with human annotations (Dinan 1 In this paper, we assume that knowledge is retrieved from documents. Kim et al., 2018) which are difficult to obtain in practice (e.g., the dataset in (Zhou et al., 2018b) does not contain annotations for knowledge selection). Therefore, we propose an unsupervised approach where learning of knowledge selection and fine-tuning of response generation are jointly conducted with unlabeled dialogues. Specifically, we build the knowledge selection module on the basis of BERT, and formalize knowledge selection as a sequence prediction process, by which the model can take advantage of the pre-training techniques and dynamically determine the relevant knowledge for a given context. The learning algorithm starts from training with pseudo ground-truth that is constructed by making full use of responses as an alternation of human annotations, and then alternatively updates the knowledge selection model and the response generation model through a reinforcement learning approach and a curriculum learning approach respectively. Thus, knowledge selection is further optimized with the feedback from response generation, and the knowledge used for fine-tuning the response generation model gradually moves from the pseudo ground-truth to the prediction of the knowledge selection module.
We test the proposed method on two benchmarks of knowledge-grounded dialogue generation: Wizard of Wikipedia (Dinan et al., 2019) and CMU Document Grounded Conversations (Zhou et al., 2018b). Evaluation results indicate that our model can significantly outperform state-of-the-art methods as well as a few pre-trained models used in heuristic ways, and thus achieves new state-of-theart on the benchmarks. Moreover, as a byproduct, the knowledge selection module also outperforms the state-of-the-art model in terms of accuracy of knowledge selection on Wizard of Wikipedia, implying that other models could also benefit from the component.
Our contributions in this paper are three-fold: (1) proposal of a knowledge selection module for applying pre-trained language models to the task of knowledge-grounded dialogue generation; (2) proposal of an unsupervised approach in which learning of knowledge selection and fine-tuning of the pre-trained model are conducted in a joint manner; and (3) empirical verification of the effectiveness of the proposed method on benchmarks of knowledge-grounded dialogue generation.
Early work on end-to-end open domain dialogue generation is inspired by the research of machine translation (Ritter et al., 2011;Shang et al., 2015;Vinyals and Le, 2015). Later, the vanilla encoderdecoder architecture is widely extended to improve diversity of responses Xing et al., 2017a;Zhao et al., 2017;Tao et al., 2018); to model the structure of conversation contexts (Serban et al., , 2017Xing et al., 2017b;Zhang et al., 2019a); to control attributes of responses Zhou et al., 2017;Zhang et al., 2018a;Wang et al., 2018;See et al., 2019); and to bias responses to some specific personas (Li et al., 2016;Zhang et al., 2018b). Recently, grounding dialogue generation by extra knowledge is emerging as an important step towards human-like conversational AI, where the knowledge could be obtained from knowledge graphs (Zhou et al., 2018a;Moon et al., 2019;Tuan et al., 2019), retrieved from unstructured documents (Dinan et al., 2019;Lian et al., 2019;Zhao et al., 2020;Kim et al., 2020), or extracted from visual background (Mostafazadeh et al., 2017;Shuster et al., 2018;Huber et al., 2018). In this work, we study document-grounded dialogue generation. Rather than learning from scratch like most existing work, we take advantage of the pre-trained language models and achieve new stateof-the-art on the benchmarks of the task.
Big, deep neural language models pre-trained on huge unlabeled text corpus have led to strong improvements on numerous natural language understanding and natural language generation benchmarks (Devlin et al., 2018;Radford et al., 2019;Song et al., 2019;Dong et al., 2019;, and therefore are revolutionizing almost the full spectrum of NLP applications (Raffel et al., 2019;Sun et al., 2019b;Qiao et al., 2019;Zhang et al., 2019b;Lample and Conneau, 2019) and some interdisciplinary applications in NLP and computer vision Su et al., 2019;Sun et al., 2019a). In the context of dialogue generation, by fine-tuning GPT-2 (Radford et al., 2019) in different sizes on social media data, recent work has (Zhang et al., 2019c;Wolf et al., 2019) shown promising progress on conversation engagement and commonsense questionanswering. In this work, we further explore the application of pre-training to the task of open domain dialogue generation by equipping the pre-trained language models with external knowledge. Differ-ent from a very recent paper on pre-training for low-resource knowledge-grounded dialogue generation (Zhao et al., 2020), the work presents an in-depth investigation on how to release the power of the existing pre-trained language models on the task when input exceeds the capacity of the models.

Problem Formalization
. . , N }, U i is a dialogue context, D i is a document that contains relevant knowledge regarding to U i , and r i is a response to U i based on D i . The goal is to learn a generation model P (r|U, D; θ) (θ denotes the parameters of the model) from D, and thus given a new dialogue context U associated with a document D, one can generate a response r following P (r|U, D; θ).

Pre-trained Language Models
We define P (r|U, D; θ) on the basis of GPT-2 from OpenAI (Radford et al., 2019). GPT-2 are transformer language models with a stack of masked multi-head self-attention layers, and are learned from large scale web text. To apply GPT-2 to the task of knowledge-grounded dialogue generation, we formulate the generation problem as P (r|U, D; θ) = P (r|g(U, D); θ) = lr t=1 P (r t |g(U, D), r 1:t−1 ; θ), (1) where g(U, D) tailors U ∪ D to meet the length constraint of a GPT-2 model as the input of generation, and r t refers to the t-th token of r whose length is supposed to be l r . The problem then boils down to (1) how to define g(U, D); and (2) how to fine-tune θ (and probably learn g(U, D)) with D.
In this work, we assume that labels that indicate the ground-truth knowledge are not available, which is practical but makes the problem even more challenging. Since D could be rather redundant with a lot of information irrelevant with the topic or the context of the conversation, simply truncating the concatenation of sentences of U and D as g(U, D) may cut the relevant knowledge and introduce noise into response generation, which hurts the performance of the GPT-2 model, as will be demonstrated in the experiments. Therefore, we consider learning a g(U, D) that can distill useful Context-aware Knowl Encoder Sequential Knowl Selector Figure 1: Architecture of the proposed model. information from D for the GPT-2 model, as will be elaborated in the next section.

Approach
Heading for learning a g(U, D) for applying GPT-2 to the task of knowledge-grounded dialogue generation, we need to deal with several challenges: (1) how to model the correlation between a context and the external knowledge; (2) how to learn g(U, D) when labels of ground-truth knowledge are absent; and (3) how to jointly optimize g(U, D) and the GPT-2 model with D, and thus the two can boost each other. Figure 1 illustrates the architecture of the model. On the basis of the transformer architecture, the knowledge selection module is made up of a context-aware knowledge encoder and a sequential knowledge selector. The former captures interaction patterns between a context U and each sentence in D through a stack of selfattention layers, and the patterns are then fed to the latter to decode useful knowledge one sentence per step. Since human annotations are not accessible, the learning method begins with pseudo groundtruth constructed by making full use of responses, and optimization of g(U, D) and optimization of the GPT-2 generation model are alternatively conducted with a reinforcement learning approach and a curriculum learning approach respectively.

Context-Aware Knowledge Encoder
We choose BERT (Devlin et al., 2018) as the backbone of the encoder. Thus, the encoder can take advantage of pre-training, and the multi-layer bidirectional attention mechanism in BERT allows a dialogue context and the associated knowledge to sufficiently interact with each other, resulting in context-aware knowledge representations. Specifically, let U = (u 1 , . . . , u n ) and D = (d 1 , . . . , d m ) be the context and the knowledge respectively, then we concatenate {u i } n i=1 as (w u 1 , · · · , w u lu ) with w u i the i-th word and l u the length of the sequence, and define the input of the encoder as S = (S 1 , . . . , S m ) with S i formulated as where w d i,j refers to the j-th word of d i ∈ D, and l d is the length of d i . Each S i ∈ S passes through the stacked self-attention layers, and is finally represented as e i = CLS(BERT(S i )) where BERT(S i ) refers to the sequence of vectors from the last layer of the encoder and CLS(·) is a function that returns the first vector of the sequence (i.e., the vector corresponding to the [CLS] token). The output of the encoder is given by E = (e 1 , . . . , e m ).

Sequential Knowledge Selector
With E as input, the sequential knowledge selector determines a subset of D (denoted as D ) as the relevant knowledge and exploits D to construct g(U, D). Since there may exist one-to-many relations between a context and the relevant knowledge (Kim et al., 2020), the size of D could vary from context to context. Therefore, we regard the construction of D as a sequence prediction process in which D starts from an empty set and gradually expands by adding one sentence from D per step. By this means, the size of D can also be viewed as a parameter and is dynamically determined according to the given context. Formally, we maintain a sequence of hidden states {s t } with the initial state s 0 a trainable parameter, and weight {d i } m i=1 by an attention mechanism which can be formulated as where W e , W s , b and v are trainable parameters. Then d jt will be added to D if j t = argmax i∈{1,...,m} P (d i |U, d j 1:t−1 ). After that, s t+1 is calculated by To determine T U,D , we introduce a special embedding e spe into E, and terminate the prediction process if e spe is selected or an upper bound T max is reached. Finally, g(U, D) is defined as the concatenation of the sentences in U ∪ D .

Learning Method
Learning a g(U, D) without human annotations is not trivial. For example, in a recent paper (Kim et al., 2020), when human labels are removed, the accuracy of knowledge selection drops from 27% to 0.3%. Moreover, since knowledge selection and response generation are entangled, ideally we hope g(U, D) and the GPT-2 model can enhance each other in learning. However, as the parameters of g(U, D) are far from optimal at the early stage, it is very possible that noise from g(U, D) will be fed to the GPT-2 model and then flows back to the learning procedure of g(U, D), resulting in inferior models on both sides. To cope with the challenges, we propose a joint optimization strategy with weak supervision as follows. The learning algorithm is summarized in Algorithm 1.
Pseudo Ground-Truth Construction. To alleviate error accumulation in joint optimization, we consider constructing weak supervision and utilize the signals to warm up the learning of g(U, D) and the fine-tuning of GPT-2 beforehand. The intuition is that responses from humans carry clues to relevance of the knowledge candidates, and thus can be used to construct pseudo ground-truth. To be specific, we first sort where Sim(·, ·) denotes a similarity function, and then build a subset of D bȳ where d j 1:t refers to the concatenation of {d j i } t i=1 . WithD, g(U, D) and the GPT-2 model are optimized via maximum likelihood estimation Joint Optimization: the Reinforcement Step.
We exploit the policy-gradient method (Sutton et al., 2000) to continue-train g(U, D) by which g(U, D) is further "supervised" by the GPT-2 model and is directly optimized for a target metric (e.g., F1 in the experiments). Specifically, we sample aD according to P (d i |U, d j 1:t−1 ) (in Eq.3.) under a termination criterion similar toD at each time step, and define the loss function as where R(D i ) = Sim(r i , r i ) with r i the response generated by the GPT-2 model given U i andD i , and b = N i=1 R(D i )/N is the baseline that is used to reduce the variance of gradient estimation (Clark and Manning, 2016). We can see that minimizing L K is equivalent to maximizing the conditional likelihood ofD i if it obtains a higher reward than the baseline.
Joint Optimization: the Curriculum Step. Though g(U, D) has been pre-trained with the pseudo ground-truthD, the relevant knowledge provided by the model (i.e., D ) may still be worse thanD at the beginning of fine-tuning. Therefore, we mix D andD and exploit a curriculum learning strategy to fine-tune the GPT-2 model where D and D are regarded as hard materials and easy materials respectively and fine-tuning gradually moves from D to D . Formally, the loss function for fine-tuning the GPT-2 model is defined by where {z i } are sampled from a Bernoulli distribution parameterized by p. By gradually shrinking p, the generation model will be exposed to more hard materials with the learning procedure going on.

6:
Update the parameters of g(U, D) based on Eq.6. the Reinforcement Step. 7: Sample {zi} from a Bernoulli distribution parameterized by p, where p = p0e −λm . 8: Update the parameters of the GPT-2 model based on Eq.7. the Curriculum Step. 9: end for 10: return g(U, D) and GPT-2.

Experiments
We conduct experiments on Wizard of Wikipedia (Wizard) and CMU Document Grounded Conversations (CMU DoG) (Zhou et al., 2018b).

Datasets and Evaluation Metrics
Both datasets are built with crowd-sourcing on Amazon Mechanical Turk, employ Wikipedia as the knowledge base, and are split into training sets, validation sets, and test sets by the data owners. Topics in Wizard cover a wide range (1, 365 in total), and each conversation happens between a wizard who has access to the knowledge about a specific topic and an apprentice who is just eager to learn from the wizard about the topic. The test set is split into two subsets: Test Seen and Test Unseen. Test Seen contains new dialogues with topics appearing in the training set, while topics in Test Unseen never appear in the training set and the validation set. We follow (Dinan et al., 2019) and conduct the pre-processing with the code published on Par-lAI 2 . Different from Wizard, CMU DoG focuses on movie domain, and besides wizard-apprentice conversations, the data also contain conversations between two workers who know the document and try to discuss the content in depth. To better compare with the baselines, we adopt the version shared at https://github.com/lizekang/ITDD. In both data, only the turns where knowledge is accessible are considered in response generation. More details are described in supplementary material.
We choose perplexity (PPL) of the ground-truth responses, BOW Embedding (Liu et al., 2016), and unigram F1 (Dinan et al., 2019)  Besides automatic evaluation, we randomly sample 300 examples from Test Seen, Test Unseen, and the test set of CMU DoG respectively, and recruit 3 well-educated native speakers as annotators for human evaluation. To each annotator, an example is presented with a context, the associated external knowledge 3 , and model responses (top 1 in greedy search) that are randomly shuffled to hide their sources. The annotators then judge the quality of the responses from three aspects, including fluency, context coherence and knowledge relevance, and assign a score in {0, 1, 2} (representing "bad", "fair", and "good") to each response for each aspect. Each response receives 3 scores per aspect, and the agreement among the annotators is measured via Fleiss' kappa (Fleiss, 1971).

Baselines
The following models are selected as baselines: Transformer Memory Network (TMN): the model proposed in (Dinan et al., 2019) along with the release of the Wizard data. We implement it using the code shared at https: //github.com/facebookresearch/ParlAI/ blob/master/projects/wizard_of_wikipedia.

Incremental Transformer with Deliberation
Decoder (ITDD): a transformer-based model ) that incrementally encodes multi-turn dialogues and knowledge and decodes responses with a deliberation technique. We implement it using the code shared at https://github.com/ lizekang/ITDD. Sequential Knowledge Transformer (SKT): a sequential latent variable model with state-of-theart performance on knowledge selection published in a very recent paper (Kim et al., 2020). Since human labels that indicate ground-truth knowl-   Disentangled Response Decoder (DRD): a model that tackles the low-resource challenge with pre-training techniques (Zhao et al., 2020). We choose the one in which all parameters are finetuned with the full training data after pre-training as the baseline, since such a configuration results in state-of-the-art performance on Wizard, as reported in (Zhao et al., 2020). We name our model KnowledGPT. Besides the baselines described above, the following pretrained models are also included in comparison in order to have a thorough understanding towards the proposed method: (1) GPT-2 trunc . We concatenate a context and the associated knowledge as a long document, and then truncate the document to meet the length constraint of the GPT-2 model. This is to check if the simple heuristics work for the task. Note that in Wizard, we randomly mix the ground-truth knowledge with others and repeat the procedure 8 times. The means with standard deviation (i.e., numbers in "( )") are reported to remove randomness; and (2) SKT+GPT-2. We feed the candidate selected by SKT to GPT-2 for response generation. This is to examine if we can simply replace the proposed knowledge selection module as well as the learning approach with an off-the-shelf knowledge selection model. Similar to SKT, the comparison is only conducted on Wizard.

Implementation Details
In both Wizard and CMU DoG, we set the hidden size and the number of layers of the sequential knowledge selector as 256 and 1 respectively. T max for D is set as 1 in Wizard, and 2 in CMU DoG. We choose BERT (110M) and GPT-2 (117M) as the pre-trained language models in KnowledGPT, and implement the models with the code in https://github.com/huggingface/ transformers. We employ greedy search in response decoding. All models are learned with Adam (Kingma and Ba, 2015) optimizer with β 1 = 0.9 and β 2 = 0.999. In warming up, we define Sim(·, ·) as unigram F1, and optimize g(U, D) and the GPT-2 model with the pseudo ground-truth for 1000 steps with a batch size of 64. In joint optimization, the batch size is set as 128, and the learning rates for g(U, D) and GPT-2 are set as 5e − 6 and 5e − 5 respectively. The learning rate will be halved if there is no improvement in terms of PPL on the validation sets. The parameter p of the Bernoulli distribution in the curriculum step is initially set as 1.0 and anneals with a rate of 1e − 5. Early stopping on validation is adopted as a regularization strategy. Table 2 and Table 3 report evaluation results on Wizard and CMU DoG respectively. KnowledGPT achieves new state-of-the-art on most metrics in both datasets, which demonstrates the effectiveness of large-scale pre-trained language models on the task of knowledge-grounded dialogue generation. GPT-2 trunc is worse than KnowledGPT, due to (1) knowledge loss: we find that in 53% test examples (Test Seen+Test Unseen), the groundtruth knowledge is cut. In this case, GPT-2 trunc only relies on the context, the related knowledge in other candidates (thanks to the one-to-many relations between a context and knowledge), and the knowledge packed in the parameters of GPT-2 for responding, which explains the comparable per-   formance with SKT and DRD; and (2) noisy input: even though the ground-truth knowledge is kept, the redundant and irrelevant information in the knowledge candidates are still harmful. Evidence is that GPT-2 trunc is worse than Knowl-edGPT on CMU DoG even though we do not cut anything on the knowledge (the maximum length of the knowledge input is 502, and thus is within the constraint of GPT-2). KnowledGPT also outperforms SKT+GPT-2 on Wizard, because (1) Knowl-edGPT is more accurate than SKT on knowledge selection, even though it does not leverage any human annotations in learning. In fact, the accuracy scores of knowledge selection for SKT are 26.8 and 18.3 on Test Seen and Test Unseen respectively, while the two numbers are 28.0 and 25.4 respectively for KnowledGPT; and (2) in KnowledGPT, knowledge selection and response generation are jointly optimized. Table 4 shows human evaluation results. While the three models are comparable on fluency, Knowl-edGPT is superior to the others on both context coherence and knowledge relevance, which is consistent with the results on automatic metrics. All kappa values are no less than 0.6, indicating substantial agreement among the annotators. We present a case study in supplementary material.

Discussions
Ablation study. To understand the impact of the learning strategies on model performance, we compare the full KnowledGPT with the following variants: (1) -pseudo: the warming up stage is removed; (2) -joint: the joint optimization stage is removed; (3) -reinforcement: g(U, D) is fixed after it is optimized with MLE on D K ; and (4) -curriculum:  GPT-2 is fixed after it is optimized with MLE on D G . Table 5 reports the evaluation results. We can conclude that (1) the pseudo ground-truth plays a crucial role in Wizard, as removing the step causes dramatic performance drop. This is because in Wizard, there is a strong correlation between the knowledge and human responses. The results indicate that though the pseudo ground-truth is constructed with heuristics, it still contains valuable information and thus allows the following joint optimization to start from a good point. On the other hand, in CMU DoG, the crowd-workers do not refer to the external knowledge as much as those workers do in Wizard when they form the responses; (2) the reinforcement step and curriculum step are useful because the reinforcement step allows the knowledge selection module to make better use of GPT-2's feedback, and through the curriculum step GPT-2 can take advantage of the output of knowledge selection module progressively; (3) joint optimization is meaningful, as removing this stage results in performance drop.
Impact of T max (i.e., the upper bound in knowledge selection). Besides the learning strategies, we are also curious about how T max , as part of the termination criterion in knowledge selection described at the end of Section 4.2, influences the performance of KnowledGPT. To this end, we vary the value of T max in {1, 2, 3} and report the evaluation results in Table 6. The larger T max is, the more chances KnowledGPT has to involve the groundtruth candidate into generation, and the lower PPL is. This also explains why the PPL of GPT-2 trunc is lower than that of KnowledGPT in Table 2 and  Table 3. On the other hand, a larger T max also means more noise in generation. That is why when T max exceeds a value, F1 begins to drop.

Conclusions
We apply large-scaled pre-trained language models to the task of knowledge-grounded dialogue generation. To this end, we devise a knowledge selection module, and propose an unsupervised approach to jointly optimizing knowledge selection and response generation. Evaluation results on two benchmarks indicate that our model can significantly outperform state-of-the-art methods.

B Comparison with DialoGPT
We compare KnowledGPT and with DialoGPT in order to learn if a pre-trained generation model with state-of-the-art performance on open domain dialogues is already good enough when it is finetuned with knowledge-grounded dialogues. We discard the associated knowledge and fine-tune DialoGPT on the knowledge-grounded dialogues. We choose the model trained from OpenAI GPT-2 with 345M parameters, as it shows the best performance in the evaluation in the original paper. The model is implemented based on the code shared at https://github.com/microsoft/DialoGPT. Table 8 shows the results, indicating that external knowledge is necessary even though one has exploited a powerful pre-trained language model for dialogue generation. In CMU DoG the gap between DialoGPT and KnowledGPT is narrowed because about 35% of the conversation has a weak correlation with the document (e.g. BLEU < 0.1).

C Impact of Maximum Tokens of GPT-2
To further justify our claims on why GPT-2 trunc is worse than KnowledGPT, we keep the groundtruth knowledge in the input sequence of GPT-2 and gradually increase the constraint of the maximum number of tokens on Wizard. As the maximum token limit increases, more irrelevant knowledge is introduced. Note that in practice, one has no way to perfectly locate the ground-truth, and this experiment is only to provide more insights to GPT-2 trunc . Table 9 shows the performance of GPT-2 trunc with the increase of the maximum   number of tokens where Ground-truth Percentage indicates the percentage of ground-truth in the input knowledge. First, when the ground-truth is forced to be kept, GPT-2 trunc is always better than the one where the ground-truth is randomly mixed with other candidates and bears the risk to be cut. This echoes our claim that knowledge loss is one of the reasons for the poor performance of GPT-D Impact of the Size of GPT-2 We further check if the performance of Knowl-edGPT can be further improved when the GPT-2 model is replaced with a larger one. Table 10 shows the results. Though GPT-2 (345M) can further reduce PPL, it does not bring significant improvement to F1 over GPT-2 (117M), probably because the larger model can not provide more accurate feedback to the knowledge selection module in learning. Therefore, to balance efficacy and cost, GPT-2 (117M) is still favored in practice. Table 11 and Table 12 show the examples from Test Seen and Test Unseen of Wizard, each example contains the dialogue context and the background knowledge which is retrieved from Wikipedia given the last two turns of dialogue and the original topic. We can see that KnowledGPT can locate the knowledge more accurately due to its knowledge selection module and reinforcement learning, and make better use of the associated knowledge with the help of curriculum learning. Knowledge (Topic: Cinematography) ... Cinematography (also called "direction of photography") is the science or art of motion-picture photography by recording light or other electromagnetic radiation, either electronically by means of an image sensor, or chemically by means of a light-sensitive material such as film stock. Typically, a lens is used to repeatedly focus the light reflected from objects into real images on the light-sensitive surface inside a camera during a questioned exposure, creating multiple images.

E Case Study
With an electronic image sensor, this produces an electrical charge at each pixel, which is electronically processed and stored in a video file for subsequent display or processing. The result with photographic emulsion is a series of invisible latent images on the film stock, which are later chemically "developed" into a visible image. The images on the film stock are played back at a rapid speed and projected onto a screen, creating the illusion of motion. Cinematography finds uses in many fields of science and business as well as for entertainment purposes and mass communication. The word "cinematography" was created from the greek words ("kinema"), meaning "movement, motion" and ("graphein") meaning "to record", together meaning "recording motion." ...

Context B
Thinking about taking a class on cinematography, the direction of photography . A I have always wanted to learn that . B Me too, I have learned its the science and or art of recording motion picture photography . A That is cool. did you enjoy it . Response TMN I think it is a great idea . It is a science fiction film that combines computer and computer science . ITDD I did . It was developed in the late 19th century . SKT Yes , I did . I loved it . It was used to focus the light reflected from objects into real images . DRD I did enjoy it . It is also called " direction of photography " . SKT+GPT-2 I did! I love how it's so easy to focus light reflected from objects into real images on the light-sensitive surface inside a camera.  Yes, I did, I love the fact that it is the science and art of recording motion picture photography KnowledGPT Yes, I did, I love cinematography, its used in many fields of science and business as well as for entertainment purposes and mass communication Human Yes and cinematography skills can also be used in mass marketing and other fields as well Knowledge(Topic: Green) ... Green is the color between blue and yellow on the visible spectrum. It is evoked by light which has a dominant wavelength of roughly 495570 nm. In subtractive color systems, used in painting and color printing, it is created by a combination of yellow and blue, or yellow and cyan; in the rgb color model, used on television and computer screens, it is one of the additive primary colors, along with red and blue, which are mixed in different combinations to create all other colors. By far the largest contributor to green in nature is chlorophyll, the chemical by which plants photosynthesize and convert sunlight into chemical energy. Many creatures have adapted to their green environments by taking on a green hue themselves as camouflage. Several minerals have a green color, including the emerald, which is colored green by its chromium content. In the middle ages and renaissance, green was the color commonly associated with merchants, bankers and the gentry, while red was reserved for the nobility. For this reason, the costume of the "mona lisa" by leonardo da vinci and the benches in the british house of commons are green while those in the house of lords are red. It also has a long historical tradition as the color of ireland and of gaelic culture. ...

Context B
Green is my favorite color. Many minerals are a green color, including emeralds. I like emeralds too! A I like green too. My eyes are green. B So are mine! In gaelic cultures green is very popular, with a long historical tradition. It is known as the color of ireland. A Yes I do know it is a very popular color in ireland. Response TMN I think it ' s a very interesting subject . It ' s a culture of culture. ITDD Yes , it is one of the primary colors . SKT It is a popular color in the us DRD It is common in nature , especially in plants . SKT+GPT-2 Yes, it is very popular in gaelic culture.  It is! It is also the second largest island in the british isles. KnowledGPT Yes, it is! In the middle ages and renaissance, green was the color commonly associated with merchants, bankers and the gentry.
Human Back in the middle ages green was often associated with merchants and bankers. Red was nobility's color of choice.