Personalizing Dialogue Agents via Meta-Learning

Existing personalized dialogue models use human designed persona descriptions to improve dialogue consistency. Collecting such descriptions from existing dialogues is expensive and requires hand-crafted feature designs. In this paper, we propose to extend Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017) to personalized dialogue learning without using any persona descriptions. Our model learns to quickly adapt to new personas by leveraging only a few dialogue samples collected from the same user, which is fundamentally different from conditioning the response on the persona descriptions. Empirical results on Persona-chat dataset (Zhang et al., 2018) indicate that our solution outperforms non-meta-learning baselines using automatic evaluation metrics, and in terms of human-evaluated fluency and consistency.


Introduction
There is a growing interest in learning personalized chit-chat dialogue agents for making chatbots more consistent. Recently, a multi-turn conversational dataset called Persona-chat (Zhang et al., 2018) has been released, where two speakers are paired and a persona description (4-5 sentences) is randomly assigned to each of them. For example, "I am an old man" and "I like to play football" are one of the possible persona descriptions provided to the speaker. By conditioning the response generation on the persona descriptions, a chit-chat model is able to produce a more persona consistent dialogue (Zhang et al., 2018).
However, it is difficult to capture a persona just by using few sentences, and collecting a nonsynthetic set of persona descriptions from a real human-human conversation, e.g., Reddit, is challenging as well since it requires hand-crafted fea- † † These two authors contributed equally. ture designs (Mazare et al., 2018). In light of this, we propose to leverage a set of dialogues done by the same persona directly, instead of using its persona descriptions, to generate a more consistent response.
We consider learning different personas as different tasks via meta-learning algorithms, which is fundamentally different from optimizing the model to represent all the personas. A high-level intuition of the difference between these two approaches is shown in Figure 1. We aim to learn a persona-independent model that is able to quickly adapt to a new persona given the dialogues. We formulate this task as a few-shot learning problem, where K dialogues are used for training and the remaining for the test. Hence, we expect to learn initial parameters of a dialogue model that can quickly adapt to the response style of a certain persona just by using few dialogues.
The main contribution of this paper is to cast the personalized dialogue learning as a meta-learning problem, which allows our model to generate personalized responses by efficiently leveraging only a few dialogue samples instead of human-designed persona descriptions. Empirical results show that our solution outperforms joint training, in terms of human-evaluated fluency and consistency.

Persona-conditioned dialogue
In Persona-chat dataset (Zhang et al., 2018), a dialogue is defined as a set of utterances U = {u 1 , . . . , u n } and a persona description is defined as a set of sentences P = {p 1 , . . . , p m }. A personalized dialogue model f θ is trained to produce a response Y = u t conditioned on previous utterances X = {u 1 , . . . , u t−1 } and persona sentences P : (1)

Persona-agnostic dialogue
Instead of conditioning our response on the persona sentences, we first adapt θ to the set of dialogue made by a persona P and then we only use the dialogue history to condition our response. Eq.
(1) becomes: Therefore, we define the set of dialogues of a persona P as D p = {U 1 , . . . , U k }. Conceptually, a model f θ is expected to generate personalized response after being trained with a few dialogues example from D p . The main idea of our work is to use Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017) to learn an initial set of parameters that can quickly learn a persona from few dialogues sample. We refer to the proposed meta-learning method for persona dialogues as Persona-Agnostic Meta-Learning (PAML).

Persona-agnostic meta-learning (PAML)
We define the persona meta-dataset as D = , the dialogue model f θ , parameterized by θ, is updated to θ p i by standard gradient descent, Algorithm for all D p i do 5: Compute adapted parameters with gradient descent: where α is learning of the inner optimization, and L D train p i the training loss. Specifically, crossentropy loss is used for training the response generation: The meta-learning model is then trained to maximize the performance of the adapted model f θ p i to the unseen dialogues in D valid p i . Following Finn et al. (2017), we define the meta-objective as: . For optimizing Eq.(5), we apply again stochastic gradient descent on the meta-model parameters θ by computing the gradient of where β is meta-learning rate. This process requires second order optimization partial derivatives, which can be computed by any automatic differentiation library (e.g. PyTorch, Tensorflow etc.). A summary of the training procedure is shown in Algorithm 1.  Table 1: Results of automatic and human evaluation: PAML vs Dialogue+Persona shows the our approach can achieve good consistency by using few dialogues instead of conditioning on the persona description, PAML vs Dialogue+Fine-tuning shows the effectiveness of meta-learning approach in personalizing dialogue model.

Experiment and Results
The experiments are conducted using Personachat (Zhang et al., 2018). To create the meta-sets D, we match the dialogues by their persona description separately for train, validation and test, by following the same persona split as in Zhang et al. (2018). On average each persona description has 8.3 unique dialogues. In the Appendix, we report the number of dialogue distribution.
Experimental setting In our experiments, we compared different training settings: (Dialogue) a model trained using dialogue history, as in Eq.
(2); (PAML) a meta-trained model as in Eq. (5), where we test each set D p i ∈ D test by selecting one dialogue and training with all the others. To elaborate, suppose we are testing U t ∈ D p i then we first fine-tuning using all the dialogues in D p i \ U t , and then test on U t . This process is repeated for all the dialogues in D p i . (Dialogue+Fine-tuning) we use the same testing as PAML but on a model trained as Dialogue. We also report a trained model that assumes persona description is available and we refer it as (Dialogue+Persona).
Implementation details We implemented f θ using a standard Transformer architecture (Vaswani et al., 2017) with pre-trained Glove embedding (Pennington et al., 2014) 1 . For the standard training, we used Adam (Kingma and Ba, 2014) optimizer with a warm-up learning rate strategy, and a batch size of 32. Instead, in meta-training, we used SGD for the inner loop and Adam for the outer loop with learning rate α = 0.01 and β = 0.0003 respectively, and batch size of 16 for both. In all the model we used beam search with beam size 5. 1 The model and the pre-processing scripts are available at https://github.com/HLTCHKUST/PAML

Evaluation metric
The objective of the evaluation is to verify whether PAML can produce a more consistent response with reference to the given dialogue and persona description (even though is not seen). To do so, we employ both automatic and human evaluation.
Automatic We report perplexity and BLEU score (Papineni et al., 2002) of the generate sentences against the human-generated prediction. Aside of standards evaluation metrics, we also train a Natural Language Inference (NLI) model using Dialog NLI (Sean et al., 2018) dataset, a recently proposed corpus based on Persona dataset, with NLI annotation between persona description sentences and dialogues utterance. We fine-tune a pre-trained BERT model (Devlin et al., 2018) using the DNLI corpus and achieve a test set accuracy of 88.43%, which is aligned to the bestreported model ESIM  in Sean et al. (2018) (with 88.20% accuracy). Then, we defined a new evaluation metric for dialogue consistency as follow: where u is a generated utterance and the p j is one sentence in the persona description. Hence, having a higher consistency C score means having a more persona consistent dialogue response.
Human Since automatic evaluation performs poorly in this task (Liu et al., 2016), we perform a human evaluation using crowd-sourced workers. We randomly selected 300 generated response examples from 10 unique personas and we asked each worker to evaluate fluency (1 to 5) and consistency of the generated response with respect to the dialogue history and the respective persona description. We asked the workers to assign a score of 1, 0 or -1 for consistent, neutral, and contradicts respectively, the full instruction set is available in the Appendix. Table 1 shows both automatic and human evaluation results. PAML achieve consistently better results in term of dialogue consistency in both automatic and human evaluation. The latter also shows that all the experimental settings have comparable fluency scores, where instead perplexity and BLEU score are lower in PAML. This confirms that these measures are not correlated to human judgment (Liu et al., 2016). For completeness, we also show generated responses examples from PAML and baseline models in Appendix.

Results
On the other hand, the human evaluated consistency is aligned to the C score, which confirms the meaningfulness of the defined measure. This agrees with results of Sean et al. (2018), where the authors showed that by re-ranking the beam search hypothesis using the DNLI score (i.e. C score), they achieved a substantial improvement in dialogue consistency.
Few-shot Learning We analyze the ability of our model to fast adapt to a certain persona in term of shots. We define shot as the number of dialogues used in D train p i for fine-tuning a certain persona, e.g. 1-shot one dialogue, 3-shot three dialogue and so on. Figure 2 compares the k-shot consistency C results for k equal to 0, 1, 3, 5 and 10, both PAML and Dialogue+Finetuning. PAML can achieve a high consistency score just by using 3 dialogues, which is better than Persona+Dialogue. On the other hand, Dialogue+Fine-tuning cannot properly leverage the dialogues in D p i , which proves the effectiveness of training with meta-learning. Meta-Learning Meta-learning (Thrun and Pratt, 1998;Schmidhuber, 1987Schmidhuber, , 1992Naik and Mammone, 1992;Bengio et al., 1992) is sub-field of machine learning with the aim of learning the learning algorithm itself. Recently, several meta-learning models has been proposed for solving few-shot image classification (Ravi and Larochelle, 2016; Vinyals et al., 2016;Finn et al., 2017;Mishra et al., 2017;Santoro et al., 2016), optimization (Andrychowicz et al., 2016) and reinforcement learning (Finn et al., 2017). Meta-learning for NLP application is less common, and it has been applied in semantic parsing task (Huang et al., 2018), machine translation for low resource language (Gu et al., 2018), and for text classification (Yu et al., 2018). To the best of our knowledge, this is the first attempt in adapting meta-learning to personalized dialogue learning.

Related Work
Personalized Dialogue Li et al. (2016) was the first to propose a persona based dialogue models for improving response consistency. Zhang et al. (2018) introduced Persona-chat, which was further extended in ConvAI2 (2019). Several works improved on the initial baselines with various methodologies (Kulikov et al., 2018;Yavuz et al.;Hancock et al., 2019;Lucas et al., 2009;Joshi et al., 2017;Zemlyanskiy and Sha, 2018;Gao et al., 2018). However, all of these previous works conditioned their response on the persona description, instead of using the dialogues produced by the persona.

Conclusion
In this paper, we present a novel meta-learning setting for personalizing dialogue agents without conditioning the model response to the persona description. This is especially useful since obtaining such persona description requires human effort. Moreover, we show that a dialogue agent trained with meta-learning achieves a more consistent dialogue by both of automatic measures and human evaluation. In future works, we plan to apply meta-learning to comment genera-tion (Lin et al., 2019) and task-oriented dialogues systems Wu et al., 2019Wu et al., , 2017Reddy et al., 2018).

Acknowledgments
This work has been funded by MRP/055/18 of the Innovation Technology Commission, of the Hong Kong University of Science and Technology.