Language Understanding for Text-based Games Using Deep Reinforcement Learning

In this paper, we consider the task of learning control policies for text-based games. In these games, all interactions in the virtual world are through text and the underlying state is not observed. The resulting language barrier makes such environments challenging for automatic game players. We employ a deep reinforcement learning framework to jointly learn state representations and action policies using game rewards as feedback. This framework enables us to map text descriptions into vector representations that capture the semantics of the game states. We evaluate our approach on two game worlds, comparing against a baseline with a bag-of-words state representation. Our algorithm outperforms the baseline on quest completion by 54% on a newly created world and by 14% on a pre-existing fantasy game.


Introduction
In this paper, we address the task of learning control policies for text-based strategy games.These games, predecessors to modern graphical ones, still enjoy a large following worldwide. 1They often involve complex worlds with rich interactions and elaborate textual descriptions of the underlying states (see Figure 1).Players read descriptions of the current world state and respond with natural language commands to take actions.Since the underlying state is not directly observable, the player has to understand the text in order to act, making it challenging for existing AI programs to play these games.
In designing an autonomous game player, we have considerable latitude when selecting an ad-State 1: The old bridge You are standing very close to the bridge's eastern foundation.If you go east you will be back on solid ground ... The bridge sways in the wind.
Command: Go east State 2: Ruined gatehouse The old gatehouse is near collapse.Part of its northern wall has already fallen down ... East of the gatehouse leads out to a small open area surrounded by the remains of the castle.There is also a standing archway offering passage to a path along the old southern inner wall.Exits: Standing archway, castle corner, Bridge over the abyss Figure 1: Sample gameplay from a Fantasy World.The player with the quest of finding a secret tomb, is currently located on an old bridge.She then chooses an action to go east that brings her to a ruined gatehouse (State 2).equate state representation to use.The simplest method is to use a bag-of-words representation derived from the text description.However, this scheme disregards the ordering of words and the finer nuances of meaning that evolve from composing words into sentences and paragraphs.For instance, in State 2 in Figure 1, the agent has to understand that going east will lead it to the castle whereas moving south will take it to the standing archway.An alternative approach is to convert text descriptions to pre-specified representations using annotated training data, commonly used in language grounding tasks (Matuszek et al., 2013;Kushman et al., 2014).
In contrast, our goal is to learn useful representations in conjunction with control policies.We adopt a reinforcement learning framework and for-mulate game sequences as Markov Decision Processes.An agent playing the game aims to maximize rewards that it obtains from the game engine upon the occurrence of certain events.The agent learns a policy in the form of an action-value function Q(s, a) which denotes the long-term merit of an action a in state s.
The action-value function is parametrized using a deep recurrent neural network, trained using the game feedback.The network contains two modules.The first one converts textual descriptions into vector representations that act as proxies for states.This component is implemented using Long Short-Term Memory (LSTM) networks (Hochreiter and Schmidhuber, 1997).The second module of the network scores the actions given the vector representation computed by the first.
We evaluate our model using two Multi-User Dungeon (MUD) games (Curtis, 1992).We compare our algorithm against baselines of a random player and a model that uses a bag-of-words representation for a state.We demonstrate that our model LSTM-DQN significantly outperforms the baselines in terms of number of completed quests and reward values.For instance, on a fantasy MUD game, our model learns to complete 96% of the quests, while the bag-of-words model and a random baseline solve only 82% and 5% of the quests, respectively.Moreover, we show that the acquired representation can be reused across games, speeding up learning and leading to faster convergence of Q-values.
Games provide a rich domain for grounded language analysis.Prior work has assumed perfect knowledge of the underlying state of the game to learn policies.Gorniak and Roy (2005) developed a game character that can be controlled by spoken instructions adaptable to the game situation.The grounding of commands to actions is learned from a transcript manually annotated with actions and state attributes.Eisenstein et al. (2009) learn game rules by analyzing a collection of game-related documents and precompiled traces of the game.In contrast to the above work, our model combines text interpretation and strategy learning in a single framework.As a result, textual analysis is guided by the received control feedback, and the learned strategy directly builds on the text interpretation.
Our work closely relates to an automatic game player that utilizes text manuals to learn strategies for Civilization (Branavan et al., 2011a).Similar to our approach, text analysis and control strategies are learned jointly using feedback provided by the game simulation.In their setup, states are fully observable, and the model learns a strategy by combining state/action features and features extracted from text.However, in our application, the state representation is not provided, but has to be inferred from a textual description.Therefore, it is not sufficient to extract features from text to supplement a simulation-based player.
Another related line of work consists of an automatic Atari games player that infers state representations directly from raw pixels (Mnih et al., 2015).This model learns control strategies using convolutional neural networks, trained with a variant of Q-learning (Watkins and Dayan, 1992).While both approaches use deep reinforcement learning for training, our work has important differences.In order to handle the sequential nature of text, we use Long Short-Term Memory networks to automatically learn useful representations.Additionally, we show that decomposing the network into a representation layer and an action selector is useful for transferring the learnt representations to new game scenarios.

Background
Game Representation We represent a game by the tuple H, A, T, R, Ψ , where H is the set of all possible game states, A = {(a, o)} is the set of all commands (action-object pairs), T (h | h, a, o) is the stochastic transition function between states and R(h, a, o) is the reward function.The game state H is hidden from the player, who only receives a varying textual description, produced by a stochastic function Ψ : H → S. Specifically, the underlying state h in the game engine keeps track of attributes such as the player's location, her health points, time of day, etc.The function Ψ (also part of the game framework) then converts this state into a textual description of the location the player is at or a message indicating low health.We do not assume access to either H or Ψ for our agent during both training and testing phases of our experiments.We denote the space of all possible text descriptions s to be S. Rewards are generated using R and are only given to the player upon completion of in-game quests.
Q-Learning Reinforcement Learning is a commonly used framework for learning control policies in game environments (Silver et al., 2007;Amato and Shani, 2010;Branavan et al., 2011b;Szita, 2012).The game environment can be formulated as a sequence of state transitions (s, a, r, s ) of a Markov Decision Process (MDP).The agent takes an action a in state s by consulting a state-action value function Q(s, a), which is a measure of the action's expected long-term reward.Q-Learning (Watkins and Dayan, 1992) is a model-free technique which is used to learn an optimal Q(s, a) for the agent.Starting from a random Q-function, the agent continuously updates its Q-values by playing the game and obtaining rewards.The iterative updates are derived from the Bellman equation (Sutton and Barto, 1998): where γ is a discount factor for future rewards and the expectation is over all game transitions that involved the agent taking action a in state s.
Using these evolving Q-values, the agent chooses the action with the highest Q(s, a) to maximize its expected future rewards.In practice, the trade-off between exploration and exploitation can be achieved following an -greedy policy (Sutton and Barto, 1998), where the agent performs a random action with probability .
Deep Q-Network In large games, it is often impractical to maintain the Q-value for all possible state-action pairs.One solution to this problem is to approximate Q(s, a) using a parametrized function Q(s, a; θ), which can generalize over states and actions by considering higher-level attributes (Sutton and Barto, 1998;Branavan et al., 2011a).However, creating a good parametrization requires knowledge of the state and action spaces.One way to bypass this feature engineering is to use a Deep Q-Network (DQN) (Mnih et al., 2015).The DQN approximates the Q-value function with a deep neural network to predict Q(s, a) for all possible actions a simultaneously given the current state s.The non-linear function layers of the DQN also enable it to learn better value functions than linear approximators.

Learning Representations and Control Policies
In this section, we describe our DQN model and describe its use in learning good Q-value approximations for games with stochastic textual descriptions.We divide the DQN model into two parts.
The first module is a representation generator that converts the textual description of the current state into a vector.This vector is then input into the second module which is an action scorer.Figure 2 shows the overall architecture of our model.We learn the parameters of both the representation generator and the action scorer jointly, using the in-game reward feedback.
Representation Generator (φ R ) The representation generator reads raw text s displayed to the agent and converts it to a vector representation v s .
A bag-of-words (BOW) representation is not sufficient to capture higher-order structures of sentences and paragraphs.The need for a better semantic representation of the text is evident from the average performance of this representation in playing MUD-games (as we show in Section 6).
In order to assimilate better representations, we utilize a Long Short-Term Memory network (LSTM) (Hochreiter and Schmidhuber, 1997) as a representation generator.LSTMs are recurrent neural networks with the ability to connect and recognize long-range patterns between words in text.They are more robust than BOW to small variations in word usage and are able to capture underlying semantics of sentences to some extent.In recent work, LSTMs have been used successfully in NLP tasks such as machine translation (Sutskever et al., 2014) and sentiment analysis (Tai et al., 2015) to compose vector representations of sentences from word-level embeddings (Mikolov et al., 2013;Pennington et al., 2014).In our setup, the LSTM network takes in word embeddings w k from the words in a description s and produces output vectors x k at each step.
To get the final state representation v s , we add a mean pooling layer which computes the elementwise mean over the output vectors x k .2 (2) Action Scorer (φ A ) The action scorer module produces scores for the set of possible actions given the current state representation.We use a multi-layered neural network for this purpose (see Figure 2).The input to this module is the vector from the representation generator, v s = φ R (s) and the outputs are scores for actions a ∈ A.
Scores for all actions are predicted simultaneously, which is computationally more efficient than scoring each state-action pair separately.Thus, by combining the representation generator and action scorer, we can obtain the approximation for the Qfunction as An additional complexity in playing MUDgames is that the actions taken by the player are multi-word natural language commands such as eat apple or go east.Due to computational constraints, in this work we limit ourselves to consider commands to consist of one action (e.g.eat) and one argument object (e.g.apple).This assumption holds for the majority of the commands in our worlds, with the exception of one class of commands that require two arguments (e.g.move red-root right, move blue-root up).We consider all possible actions and objects available in the game and predict both for each state using the same network (Figure 2).We consider the Q-value of the entire command (a, o) to be the average of the Qvalues of the action a and the object o.For the rest of this section, we only show equations for Q(s, a) but similar ones hold for Q(s, o).
Parameter Learning We learn the parameters θ R of the representation generator and θ A of the action scorer using stochastic gradient descent with RMSprop (Tieleman and Hinton, 2012).The complete training procedure is shown in Algorithm 1.In each iteration i, we update the parameters to reduce the discrepancy between the predicted value of the current state Q(s t , a t ; θ i ) (where θ i = [θ R ; θ A ] i ) and the expected Q-value given the reward r t and the value of the next state max a Q(s t+1 , a; θ i−1 ).
We keep track of the agent's previous experiences in a memory D.3 Instead of performing updates to the Q-value using transitions from the current episode, we sample a random transition (ŝ, â, s , r) from D. Updating the parameters in this way avoids issues due to strong correlation when using transitions of the same episode (Mnih et al., 2015).Using the sampled transition and (1), we obtain the following loss function to minimize: (3) where is the target Q-value with parameters θ i−1 fixed from the previous iteration.
The updates on the parameters θ can be performed using the following gradient of L i (θ i ): For each epoch of training, the agent plays several episodes of the game, which is restarted after every terminal state.

Mini-batch Sampling
In practice, online updates to the parameters θ are performed over a mini batch of state transitions, instead of a single transition.This increases the number of experiences used per step and is also more efficient due to optimized matrix operations.
The simplest method to create these minibatches from the experience memory D is to sample uniformly at random.However, certain experiences are more valuable than others for the agent to learn from.For instance, rare transitions Sample random mini batch of transitions (s j , a j , r j , s j+1 , p j ) from D, with fraction ρ having p j = 1 16: Set Perform gradient descent step on the loss L(θ) = (y j − Q(s j , a j ; θ)) 2 that provide positive rewards can be used more often to learn optimal Q-values faster.In our experiments, we consider such positive-reward transitions to have higher priority and keep track of them in D. We use prioritized sampling (inspired by Moore and Atkeson (1993)) to sample a fraction ρ of transitions from the higher priority pool and a fraction 1 − ρ from the rest.

Experimental Setup
Game Environment For our game environment, we modify Evennia, 4 an open-source library for building online textual MUD games.Evennia is a Python-based framework that allows one to easily create new games by writing a batch file describing the environment with details of rooms, objects and actions.The game engine keeps track of the game state internally, presenting textual descriptions to the player and receiving text commands from the player.We conduct experiments on two worlds -a smaller Home world we created ourselves, and a larger, more complex Fantasy world created by Evennia's developers.The motivation behind Home world is to abstract away high-level planning and focus on the language understanding requirements of the game.In both worlds, the agent gets a randomly sampled description of a room each time.The agent receives a positive reward on completing a quest, and negative rewards for getting into bad situations like falling off a bridge, or losing a battle.We also add small deterministic negative rewards for each non-terminating step.This incentivizes the agent to learn policies that solve quests in fewer steps.Supplementary material has reward details.
Home World We created Home world to mimic the environment of a typical house. 5The world consists of four rooms -a living room, a bedroom, a kitchen and a garden with connecting pathways.Every room is reachable from every other room.Each room contains a representative object that the agent can interact with.For instance, the kitchen has an apple that the player can eat.Transitions between the rooms are deterministic.At the start of each game episode, the player is placed in a random room and provided with a randomly selected quest.The text provided to the player contains both the description of her current state and that of the quest.Thus, the player can begin in one of 16 different states (4 rooms × 4 quests), which adds to the world's complexity.
An example of a quest given to the player in text is You are not sleepy but are hungry.To complete this quest and obtain a reward, the player has to navigate through the house to reach the kitchen and eat the apple (i.e type in the command eat apple).More importantly, the player should interpret that the quest does not require him to take a nap in the bedroom.We created such misguiding quests to make it hard for agents to succeed without having an adequate level of language understanding.

Fantasy World
The Fantasy world is considerably more complex and involves quests such as navigating through a broken bridge or finding the secret tomb of an ancient hero.This game also has stochastic transitions in addition to varying state descriptions provided to the player.For instance, there is a possibility of the player falling from the bridge if he lingers too long on it.
Due to the large command space in this game,6 we make use of cues provided by the game itself to narrow down the set of possible objects to consider in each state.For instance, in the MUD example in Figure 1, the game provides a list of possible exits.If the game does not provide such clues for the current state, we consider all objects in the game.
Evaluation We use two metrics for measuring an agent's performance: (1) the cumulative reward obtained per episode averaged over the episodes and (2) the fraction of quests completed by the agent.The evaluation procedure is as follows.In each epoch, we first train the agent on M episodes of T steps each.At the end of this training, we have a testing phase of running M episodes of the game for T steps.We use M = 50, T = 20 for the Home world and M = 20, T = 250 for the Fantasy world.For all evaluation episodes, we run the agent following an -greedy policy with = 0.05, which makes the agent choose the best action according to its Q-values 95% of the time.We report the agent's performance at each epoch.
Baselines We compare our LSTM-DQN model to two baselines.The first is a Random agent that chooses both actions and objects uniformly at random from all available choices. 7The second is BOW-DQN, which uses a bag-of-words representation of the text as input to the DQN action scorer.This baseline serves to illustrate the importance of having a good representation layer for the task.
Settings For our DQN models, we used D = 100000, γ = 0.5.We use a learning rate of 0.0005 for RMSprop.We anneal the for -greedy from 1 to 0.2 over 100000 transitions.We perform a minibatch gradient update every 4 steps of the gameplay.We roll out the LSTM (over words) for a maximum of 30 steps on the Home world and for 100 steps on the Fantasy world.For the prioritized sampling, we used ρ = 0.25 for both worlds.We used a mini-batch size of 64 and embedding size d = 20 in all experiments.

Results
Home World Figure 3 illustrates the performance of LSTM-DQN compared to the baselines.We can observe that the Random baseline performs quite poorly, completing only around 10% of quests on average8 obtaining a low reward of around −1.58.The BOW-DQN model performs significantly better and is able to complete around 46% of the quests, with an average reward of 0.20.The improvement in reward is due to both greater quest success rate and a lower rate of issuing invalid commands (e.g.eat apple would be invalid in the bedroom since there is no apple).We notice that both the reward and quest completion graphs of this model are volatile.This is because the model fails to pick out differences between quests like You are not hungry but are sleepy and You are not sleepy but are hungry.In contrast, the LSTM-DQN model does not suffer from this issue and is able to complete 100% of the quests after around 50 epochs of training, achieving close to the optimal reward possible.9 Fantasy World We evaluate all the models on the Fantasy world in the same manner as before and report reward, quest completion rates and Qvalues.The quest we evaluate on involves crossing the broken bridge (which takes a minimum of five steps), with the possibility of falling off at random (a 5% chance) when the player is on the bridge.The game has an additional quest of reaching a secret tomb.However, this quest is complex since it requires the player to memorize game events and perform high-level planning that are beyond the scope of this current work.Therefore, we focus only on the first quest.
From Figure 3 (bottom), we can see that the Random baseline does poorly in terms of both average per-episode reward 10 and quest completion rates.BOW-DQN converges to a much higher average reward of −12.68 and achieves around 82% quest completion.Again, the BOW-DQN is often confused by varying (10 different) descriptions of the portions of the bridge, which reflects in its erratic performance on the quest.The LSTM-DQN model again outperforms both baselines, achieving an average reward of −11.33 and completing 96% of quests.Though this world does not contain descriptions adversarial to BOW-LSTM, the LSTM-DQN obtains higher average reward by completing the quest in fewer steps by being more 10 Note that the rewards graph is in log scale.resilient to variations in the state descriptions.
Transfer Learning We would like the representations learnt by φ R to be generic enough and transferable to new game worlds.To test this, we created a second Home world with the same rooms, but a completely different map, changing the locations of the rooms and the pathways between them.The main differentiating factor of this world from the original home world lies in the high-level planning required to complete quests.
We initialized the LSTM part of an LSTM-DQN agent with parameters θ R learnt from the original home world and trained it on the new world.11Figure 3 (top right) demonstrates that the agent with transferred parameters is able to learn quicker than an agent starting from scratch initialized with random parameters (No Transfer), reaching the optimal policy almost 20 epochs earlier.This indicates that these simulated worlds can be used to learn good representations for language that transfer across worlds.Table 2: Sample descriptions from the Fantasy world and their nearest neighbors (NN) according to their vector representations from the LSTM representation generator.The NNs are often descriptions of the same or similar (nearby) states in the game.

Prioritized sampling
on the parameter learning.From Figure 3 (bottom right), we observe that using prioritized sampling significantly speeds up learning, with the agent achieving the optimal policy around 50 epochs faster than using uniform sampling.This shows promise for further research into different schemes of assigning priority to transitions.

Representation Analysis
We analyzed the representations learnt by the LSTM-DQN model on the Home world.Figure 4 shows a visualization of learnt word embeddings, reduced to two dimensions using t-SNE (Van der Maaten and Hinton, 2008).All the vectors were initialized randomly before training.We can see that semantically similar words appear close together to form coherent subspaces.In fact, we observe four different subspaces, each for one type of room along with its corresponding object(s) and quest words.For instance, food items like pizza and rooms like kitchen are very close to the word hungry which appears in a quest description.This shows that the agent learns to form meaningful associations between the semantics of the quest and the environment.Table 2 shows some examples of descriptions from Fantasy world and their nearest neighbors using cosine similarity between their corresponding vector representations produced by LSTM-DQN.The model is able to correlate descriptions of the same (or similar) underlying states and project them onto nearby points in the representation subspace.

Conclusions
We address the task of learning control policies for text-based games.In these games, all interactions in the virtual world are through text and the underlying state is not observed.The resulting language barrier makes such environments chal-  lenging for automatic game players.We employ a deep reinforcement learning framework to jointly learn state representations and action policies using game rewards as feedback.This framework enables us to map text descriptions into vector representations that capture the semantics of the game states.We evaluate our approach on two game worlds, comparing against a baseline with a bagof-words state representation.Our algorithm outperforms the baseline on quest completion by 54% on a newly created world and by 14% on a preexisting fantasy game.Future directions include tackling high-level planning and strategy learning to improve the performance of intelligent agents.

Figure 2 :
Figure 2: Architecture of LSTM-DQN: The Representation Generator (φ R ) (bottom) takes as input a stream of words observed in state s and produces a vector representation v s , which is fed into the action scorer (φ A ) (top) to produce scores for all actions and argument objects.

Figure 3 :
Figure 3: Left: Graphs showing the evolution of average reward and quest completion rate for BOW-DQN, LSTM-DQN and a Random baseline on the Home world (top) and Fantasy world (bottom).Note that the reward is shown in log scale for the Fantasy world.Right: Graphs showing effects of transfer learning and prioritized sampling on the Home world.
We also investigate the effects of different minibatch sampling proceduresDescriptionNearest neighborYou are halfways out on the unstable bridge.From the castle you hear a distant howling sound, like that of a large dog or other beast.The bridge slopes precariously where it extends westwards towards the lowest point -the center point of the hang bridge.You clasp the ropes firmly as the bridge sways and creaks under you.The ruins opens up to the sky in a small open area, lined by columns.... To the west is the gatehouse and entrance to the castle, whereas southwards the collumns make way for a wide open courtyard.The old gatehouse is near collapse..... East the gatehouse leads out to a small open area surrounded by the remains of the castle.There is also a standing archway offering passage to a path along the old southern inner wall.

Figure 4
Figure 4: t-SNE visualization of the word embeddings (except stopwords) after training on Home world.The embedding values are initialized randomly.
Algorithm 1 Training Procedure for DQN with prioritized sampling 1: Initialize experience memory D 2: Initialize parameters of representation generator (φ R ) and action scorer (φ A ) randomly 3: for episode = 1, M do Store transition (s t , a t , r t , s t+1 , p t ) in D

Table 1 :
Table 1 provides statistics of the game worlds.Various statistics of the two game worlds 4 http://www.evennia.com/