Deep Reinforcement Learning for Dialogue Generation

Recent neural models of dialogue generation offer great promise for generating responses for conversational agents, but tend to be shortsighted, predicting utterances one at a time while ignoring their influence on future outcomes. Modeling the future direction of a dialogue is crucial to generating coherent, interesting dialogues, a need which led traditional NLP models of dialogue to draw on reinforcement learning. In this paper, we show how to integrate these goals, applying deep reinforcement learning to model future reward in chatbot dialogue. The model simulates dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity (non-repetitive turns), coherence, and ease of answering (related to forward-looking function). We evaluate our model on diversity, length as well as with human judges, showing that the proposed algorithm generates more interactive responses and manages to foster a more sustained conversation in dialogue simulation. This work marks a first step towards learning a neural conversational model based on the long-term success of dialogues.


Introduction
Neural response generation (Li et al., 2015;Vinyals and Le, 2015;Luan et al., 2016;Wen et al., 2015;Shang et al., 2015;Yao et al., 2015;Xu et al., 2016;Wen et al., 2016;Li et al., 2016;Su et al., 2016) is of growing interest.The LSTM sequence-to-sequence (SEQ2SEQ) model (Sutskever et al., 2014) is one type of neural generation model that maximizes the probability of generating a response given the previous dialogue turn.This approach enables the incorporation of rich context when mapping between consec-utive dialogue turns (Sordoni et al., 2015) in a way not possible, for example, with MT-based dialogue models (Ritter et al., 2011).
Despite the success of SEQ2SEQ models in dialogue generation, two problems emerge: First, SEQ2SEQ models are trained by predicting the next dialogue turn in a given conversational context using the maximum-likelihood estimation (MLE) objective function.However, it is not clear how well MLE approximates the real-world goal of chatbot development: teaching a machine to converse with humans, while providing interesting, diverse, and informative feedback that keeps users engaged.One concrete example is that SEQ2SEQ models tend to generate highly generic responses such as"I don't know" regardless of the input (Sordoni et al., 2015;Serban et al., 2015b;Serban et al., 2015c;Li et al., 2015).This can be ascribed to the high frequency of generic responses found in the training set and their compatibility with a diverse range of conversational contexts.Apparently "I don't know" is not a good action to take, since it closes the conversation down.
Another common problem, illustrated in Table 1 (the example in the bottom left), is when the system becomes stuck in an infinite loop of repetitive responses.This is due to MLE-based SEQ2SEQ models' inability to account for repetition.In example 2, the dialogue falls into an infinite loop after three turns, with both agents generating dull, generic utterances like i don't know what you are talking about and you don't know what you are saying.Looking at the entire conversation, utterance (2) i'm 16 turns out to be a bad action to take.While it is an informative and coherent response to utterance (1) asking about age, it offers no way of continuing the conversation. 1aseline mutual information model (Li et al. 2015)  ) is input by the authors.Then the two agents take turns conversing, taking as input the other agent's prior generated turn.The output is generated using the mutual information model (Li et al., 2015) in which an N-best list is first obtained using beam search based on p(t|s) and reranked by linearly combining the backward probability p(s|t), where t and s respectively denote targets and sources.Right Column: Dialogue simulated using the proposed reinforcement learning model.The new model has more forward-looking utterances (questions like "Why are you asking?" and offers like "I'll come with you") and lasts longer before it falls into conversational black holes.
These challenges suggest we need a conversation framework that has the ability to (1) integrate developer-defined rewards that better mimic the true goal of chatbot development and (2) model the longterm influence of a generated response in an ongoing dialogue.
To achieve these goals, we draw on the insights of reinforcement learning, which have been widely applied in MDP and POMDP dialogue systems (see Related Work section for details).We introduce a neural reinforcement learning (RL) generation method, which can optimize long-term rewards designed by system developers.Our model uses the encoderdecoder architecture as its backbone, and simulates conversation between two virtual agents to explore the space of possible actions while learning to maximize expected reward.We define simple heuristic approximations to rewards that characterize good conversations: good conversations are forward-looking (Allwood et al., 1992) or interactive (a turn suggests a following turn), informative, and coherent.The parameters of an encoder-decoder RNN define a policy over an infinite action space consisting of all possible utterances.The agent learns a policy by optimizing the long-term developer-defined reward from ongoing dialogue simulations using policy gradient methods (Williams, 1992), rather than the MLE objective defined in standard SEQ2SEQ models.
Our model thus integrates the power of SEQ2SEQ systems to learn compositional semantic meanings of utterances with the strengths of reinforcement learning in optimizing for long-term goals across a conversation.Experimental results (sampled results at the right panel of Table 1) demonstrate that our approach fosters a more sustained dialogue and manages to produce more interactive responses than standard SEQ2SEQ models trained using the MLE objective.

Related Work
Efforts to build statistical dialog systems fall into two major categories.
The first treats dialogue generation as a sourceto-target transduction problem and learns mapping rules between input messages and responses from a massive amount of training data.Ritter et al. (2011) frames the response generation problem as a statistical machine translation (SMT) problem.Sordoni et al. (2015) improved Ritter et al.'s system by rescor-ing the outputs of a phrasal SMT-based conversation system with a neural model that incorporates prior context.Recent progress in SEQ2SEQ models inspire several efforts (Vinyals and Le, 2015) to build endto-end conversational systems which first apply an encoder to map a message to a distributed vector representing its semantics and generate a response from the message vector.Serban et al. (2015a) propose a hierarchical neural model that captures dependencies over an extended conversation history.Li et al. (2015) propose mutual information between message and response as an alternative objective function in order to reduce the proportion of generic responses produced by SEQ2SEQ systems.
The other line of statistical research focuses on building task-oriented dialogue systems to solve domain-specific tasks.Efforts include statistical models including Markov Decision Processes (MDPs) (Levin et al., 1997;Levin et al., 2000;Walker et al., 2003;Pieraccini et al., 2009), POMDP (Young et al., 2010;Young et al., 2013) models, and models that statistically learn generation rules (Oh and Rudnicky, 2000;Ratnaparkhi, 2002;Banchs and Li, 2012;Nio et al., 2014).This dialogue literature thus widely appliess reinforcement learning (Walker, 2000;Schatzmann et al., 2006;Gasic et al., 2013;Singh et al., 1999;Singh et al., 2000;Singh et al., 2002) to train dialogue policies.But task-oriented RL dialogue systems often rely on carefully limited dialogue parameters, or hand-built templates with state, action and reward signals designed by humans for each new domain, making the paradigm difficult to extend to open-domain scenarios.
Also relevant is prior work on reinforcement learning for language understanding -including learning from delayed reward signals by playing text-based games (Narasimhan et al., 2015;He et al., 2015), executing instructions in windows help interacting with game environments (Branavan et al., 2011).
Our goal is to integrate these two paradigms, drawing on the advantages of both.We are thus inspired by recent work like Wen et al. (2016) trains an endto-end task-oriented dialogue system that links input representations to slot-value pairs in a database.Also relevant is a recent work in Su et al. ( 2016) that adopted a two-stage training using gradient-based methods by combining reinforcement learning with neural generation models.Su et al. (2016) explored the combine model on tasks that involves the interaction with real user and show the reinforcement learning helps the dialogue performance.

Reinforcement Learning for
Open-Domain Dialogue In this section, we describe in detail the components of the proposed RL model.
The learning system consists of two agents.We use p to denote sentences generated from the first agent and q to denote sentences from the second.The two agents take turns talking with each other.A dialogue can be represented as a alternating sequence of sentences generated by the two agents: p 1 , q 1 , p 2 , q 2 , ..., p i , q i .We view the generated sentences as actions that are taken according to a policy defined a by an encoder-decoder recurrent neural network language model.
The parameters of the network are optimized to maximize expected future reward using policy search, as described in Section 4.3.Policy gradient methods are more appropriate for our scenario than Q-learning (Mnih et al., 2013), because we can initialize the encoder-decoder RNN using MLE parameters that already produce plausible responses, before changing the objective and tuning towards a policy that maximizes long-term reward.Q-learning, on the other hand, directly estimates the future expected reward of each action, which can differ from the MLE objective by orders of magnitude, thus making MLE parameters inapropriate for initialization.The components (states, actions, reward, etc.) of our sequential decision problem are summarized in the following sub-sections.

Action
An action a is the dialogue utterance to generate.The action space is infinite since arbitrary-length sequences can be generated.

State
A state is denoted by the previous two dialogue turns [p i , q i ].The dialogue history is further transformed to a vector representation by feeding the concatenation of p i and q i into an LSTM encoder model as described in Li et al. (2015).

Policy
A policy takes the form of an LSTM encoder-decoder (i.e., p RL (p i+1 |p i , q i ) ) and is defined by its parameters.Note that we use a stochastic representation of the policy (a probability distribution over actions given states).A deterministic policy would result in a discontinuous objective that is difficult to optimize using gradient-based methods.

Reward
r denotes the reward obtained for each action.In this subsection, we discuss major factors that contribute to the success of a dialogue and describe how approximations to these factors can be operationalized in computable reward functions.
Ease of answering A turn generated by a machine should be easy to respond to.This aspect of a turn is related to its forward-looking function: the constraints a turn places on the next turn (Schegloff and Sacks, 1973;Allwood et al., 1992).We propose to measure the ease of answering a generated turn by using the negative log likelihood of reponding to that utterance with a dull response.We manually constructed a list of dull responses S consisting 8 turns such as "I don't know what you are talking about", "I have no idea", etc., that we and others have found occur very frequently in SEQ2SEQ models of conversations.The reward function is given as follows: where N S denotes the cardinality of N S and N s denotes the number of tokens in the dull response s.
Although of course there are more ways to generate dull responses than the list can cover, many of these responses are likely to fall into similar regions in the vector space computed by the model.A system less likely to generate utterances in the list is thus also less likely to generate other dull responses.p seq2seq represents the likelihood output by SEQ2SEQ models.It is worth noting that p seq2seq is different from the stochastic policy function p RL (p i+1 |p i , q i ), since the former is learned based on the MLE objective of the SEQ2SEQ model while the latter is the policy optimized for long-term future reward in the RL setting.r 1 is further scaled by the length of target S.
Information Flow We want each agent to contribute new information at each turn to keep the dialogue moving and avoid repetitive sequences.We therefore propose penalizing semantic similarity between consecutive turns from the same agent.Let h p i and h p i+1 denote representations obtained from the encoder for two consecutive turns p i and p i+1 .The reward is given by the negative log of the cosine similarity between them: Semantic Coherence We also need to measure the adequacy of responses to avoid situations in which the generated replies are highly rewarded but are ungrammatical or not coherent.We therefore consider mutual information between the action a and previous turns in the history to ensure the generated responses are coherent and appropriate: ) p seq2seq (a|p i , q i ) denotes the probability of generating response a the given previous dialogue utterances [p i , q i ]. p backward seq2seq (q t |a) denotes the backward probability of generating the previous dialogue utterance q t based on response a. p backward seq2seq is trained in a similar way as standard SEQ2SEQ models with sources and targets swapped.Again, to control for the influence of target length, both log p seq2seq (a|q i , p i ) and log p backward seq2seq (q i |a) are scaled by the length of targets.The final reward for action a is a weighted sum of the rewards discussed above: where λ 1 + λ 2 + λ 3 = 1.We set λ 1 = 0.25, λ 2 = 0.25 and λ 3 = 0.5.A reward is observed after the agent reaches the end of each sentence.

Simulation
The central idea behind our approach is to simulate the process of two virtual agents taking turns talking with each other, through which we can explore the state space and learn a policy p RL (p i+1 |p i , q i ) that leads to optimal expected reward.

Supervised Learning
For the first stage of training, we build on prior work of predicting a generated target sequence given dialogue history from the supervised SEQ2SEQ model (Vinyals and Le, 2015).Results from supervised models will be later used for initialization.
We trained a SEQ2SEQ model with attention (Bahdanau et al., 2014) on the Opensubtitle dataset, which consists of roughly 80 million source-target pairs.We treated each turn in the dataset as a target and the concatenation of two previous sentences as source inputs.

Mutual Information
Samples from SEQ2SEQ models are often times dull and generic, e.g., "i don't know" (Li et al., 2015) We thus do not want to initialize the policy model using the pre-trained SEQ2SEQ models because this will lead to a lack of diversity in the RL models' experiences.Li et al. (2015) showed that modeling mutual information between sources and targets will significantly decrease the chance of generating dull responses and improve general response quality.We now show how we can obtain an encoder-decoder model which generates maximum mutual information responses.
As illustrated in Li et al. (2015), direct decoding from Eq 3 is infeasible since the second term requires the target sentence to be completely generated.Inspired by recent work on sequence level learning (Ranzato et al., 2015), we treat the problem of generating maximum mutual information response as a reinforcement learning problem in which a reward of mutual information value is observed when the model arrives at the end of a sequence.
Similar to Ranzato et al. ( 2015), we use policy gradient methods (Sutton et al., 1999;Williams, 1992) for optimization.We initialize the policy model p RL using a pre-trained p SEQ2SEQ (a|p i , q i ) model.Given an input source [p i , q i ], we generate a candidate list A = {â|â ∼ p RL }.For each generated candidate â, we will obtain the mutual information score m(â, [p i , q i ]) from the pre-trained p SEQ2SEQ (a|p i , q i ) and p backward SEQ2SEQ (q i |a).This mutual information score will be used as reward and back propagate to the encoder-decoder model, tailoring it to generate sequences with higher rewards.We refer the readers to Zaremba and Sutskever (2015) and Williams (1992) for details.The expected reward for a sequence is given by: (5) The gradient is estimated using the likelihood ratio trick: We update the parameters in the encoder-decoder model using stochastic gradient descent.A curriculum learning strategy is adopted (Bengio et al., 2009) as in (Ranzato et al., 2015) such that, for every sequence of length T we use the MLE loss for the first L tokens and reinforce algorithm for the rest T − L tokens.We gradually anneal the value of L to zero.Baseline strategy is employed to decrease the learning variance.For simplicity, we set the baseline b to a fixed value of −3 which approximates the average MMI reward score.This makes final gradient as follows:

Dialogue Simulation between Two Agents
We simulate conversation between the two virtual agents and have them take turns talking with each other.The simulation proceeds as follows: at the initial step, a message from the training set is fed to the first agent.The agent encodes the input message to a vector representation and starts decoding to generate a response output.Combining the immediate output from the first agent with the dialogue history, the second agent updates the state by encoding the dialogue history into a representation and uses the decoder RNN to generate responses, which are subsequently fed back to the first agent, and the process is repeated.
Optimization We initialize the policy model p RL with parameters from the mutual information model described in the previous subsection.We then use policy gradient methods to find parameters that lead to a large expected reward.For a sequence of actions {a 1 , a 2 , ..., a T }, the reward function is given by: p 1,3 Turn 2 . . .where R(a i , [p i , q i ]) denotes reward resulting from action a i .Again, the reinforce algorithm (Williams, 1992) is used for gradient updates: (9) We refer readers to Williams (1992) and Glynn (1990) for more details.For each simulated instance, we generate 5 candidate for each dialogue term.Since most of the generated translations in the N-best list are similar, differing only by punctuation or minor morphological variations, candidates are generated by sampling from the distribution with additional Gaussian noise to foster more diverse candidates.

Curriculum Learning
Curriculum Learning strategy is again employed in which we begin with simulating the dialogue for 2 turns, and gradually increase the number of simulated turns.We generate 5 turns at most, as the number of candidates to examine grows exponentially in the size of candidate list.Five candidate responses are generated at each step of the simulation.

Experimental Results
In this section, we describe experimental results along with qualitative analysis.We evaluate dialogue generation systems using both human judgments and two automatic metrics: conversation length and diversity.

Dataset
The dialogue simulation requires high-quality initial inputs fed to the agent.For example, an initial input of "why ?" is undesirable since it is unclear how the dialogue could proceed.We take a subset of 10 million messages from the Opensubtitle dataset and extract 0.8 million sequences with the lowest likelihood of generating the response "i don't know what you are taking about" to make ensure initial inputs are easy to respond to.

Automatic Evaluation
Evaluating dialogue systems is difficult.Metrics such as BLEU (Papineni et al., 2002) and perplexity have been widely used for dialogue quality evaluation (Li et al., 2015;Vinyals and Le, 2015;Sordoni et al., 2015), but it is widely debated how well these automatic metrics are correlated with true response quality (Liu et al., 2016;Galley et al., 2015).Since the goal of the proposed system is not to predict the highest probability response, but rather the long-term success of the dialogue, we do not employ BLEU or perplexity for evaluation 2 . 2 We found the RL model performs worse on BLEU score.On a random sample of 2,500 conversational pairs, single reference BLEU scores for RL models, mutual information models and vanilla SEQ2SEQ models are respectively 1.28, 1.44 and 1.17.BLEU is highly correlated with perplexity in generation tasks.Since the RL model is trained based on future reward rather than Model # of simulated turns SEQ2SEQ 2.68 mutual information 3.40 RL 4.48

Length of the dialogue
The first metric we propose is the length of the simulated dialogue.We say a dialogue ends when one of the agents starts generating dull responses such as "i don't know"3 or two consecutive utterances from the same user are highly overlapping4 .The test set consists of 1,000 input messages.To reduce the risk of circular dialogues, we limit the number of simulated turns to be less than 8. Results are shown in Table 2.As can be seen, using mutual information leads to more sustained conversations between the two agents.The proposed RL model is first trained based on the mutual information objective and thus benefits from it in addition to the RL model.We observe that the RL model with dialogue simulation achieves the best evaluation score.
Diversity We report degree of diversity by calculating the number of distinct unigrams and bigrams in generated responses.The value is scaled by the total number of generated tokens to avoid favoring long sentences as described in (Li et al., 2015).The resulting metric is thus a type-token ratio for unigrams and bigrams.
For both the standard SEQ2SEQ model and the proposed RL model, we use beam search with a beam size 10 to generate a response to a given input message.For the mutual information model, we first generate n-best lists using p SEQ2SEQ (t|s) and then linearly re-rank them using p SEQ2SEQ (s|t).Results are presented in Table 4.We find that the proposed RL model generates more diverse outputs when compared against both the vanilla SEQ2SEQ model and

Human Evaluation
We explore three settings for human evaluation: the first setting is similar to what was described in Li et al. (2015), where we employ crowdsourced judges to evaluate a random sample of 500 items.We present both an input message and the generated output to 3 judges and ask them to decide which of the two outputs is better (denoted as singleturn general quality).Ties are permitted.Identical strings are assigned the same score.We measure the improvement achieved by the RL model over the mutual information model by the mean difference in scores between the models.For the second setting, judges are again presented with input messages and system outputs, but are asked to decide which of the two outputs is easier to respond to (denoted as single-turn ease to answer).Again we evaluate a random sample of 500 items, each being assigned to 3 judges.
For the third setting, judges are presented with simulated conversations between the two agents (denoted as multi-turn general quality).Each conversation consists of 5 turns.We evaluate 200 simulated conversations, each being assigned to 3 judges, who are asked to decide which of the simulated conversations is of higher quality.
Results for human evaluation are shown in Table 5.The proposed RL system does not introduce a significant boost in single-turn response quality with 95% CT [-0.02, 0.05], which is in line with our ex-  pectation, as the RL model is not optimized to predict the next utterance, but rather to increase long-term reward.The RL system produces responses that are significantly easier to answer than mutual information, as demonstrated by the single-turn ease to answer setting, and also higher quality multi-turn dialogues, as demonstrated by the multi-turn general quality setting.

Qualitative Analysis and Discussion
We show a random sample of generated responses in Table 3 and simulated conversations in Table 1 at the beginning of the paper.From Table 3, we can see that the RL based agent indeed generates more interactive responses than the other baselines.We also find that the RL model has a tendency to end a sentence with another question and hand the conversation over to the user.From Table 1, we observe that the RL model manages to produce more interactive and sustained conversations than the mutual information model.During error analysis, we found that although we penalize repetitive utterances in consecutive turns, the dialogue sometimes enters a cycle with length greater than one, as shown in Table 6.This can be ascribed to the limited amount of conversational history we consider.Another issue observed is that the model sometimes starts a less relevant topic during the conversation.There is a tradeoff between relevance and less repetitiveness, as manifested in the reward function we define in Eq 4.
The fundamental problem, of course, is that the manually defined reward function can't possibly cover the crucial aspects that define an ideal conversation.While the heuristic rewards that we defined are amenable to automatic calculation, and do capture some aspects of what makes a good conversation, ideally the system would instead receive real rewards from humans.Another problem with the current model is that we can only afford to explore a very small number of candidates and simulated turns since the number of cases to consider grow exponentially.

Conclusion
We introduce a reinforcement learning framework for neural response generation by simulating dialogues between two agents, integrating the strengths of neural SEQ2SEQ systems and reinforcement learning for dialogue.Like earlier neural SEQ2SEQ models, our framework captures the compositional models of the meaning of a dialogue turn and generates semantically appropriate responses.Like reinforement learning dialogue systems, our framework is able to generate utterances that optimize future reward, successfully capturing global properties of a good conversation.Despite the fact that model uses very simple, operationable heuristics for capturing these global properties, the framework generates more diverse, interactive responses that foster a more sustained conversation.

Figure 1 :
Figure 1: Dialogue simulation between the two agents.

Table 1 :
Left Column: Dialogue simulation between two agents using a 4-layer LSTM encoder-decoder trained on the Opensubtitle dataset.The first turn (index 1

Table 2 :
The average number of simulated turns from standard SEQ2SEQ models, mutual information model and the proposed RL model.
Do you have any feelings for me ?I don't know what you are talking about.I don 't want to hurt your feelings .How much time do you have here?Not long enough.Sorry, sir.

Table 3 :
Sampled responses generated from the mutual information models and the proposed RL model.

Table 6 :
An simulated dialogue with a cycle longer than one.