A Survey of Text Games for Reinforcement Learning Informed by Natural Language

Abstract Reinforcement Learning has shown success in a number of complex virtual environments. However, many challenges still exist towards solving problems with natural language as a core component. Interactive Fiction Games (or Text Games) are one such problem type that offer a set of safe, partially observable environments where natural language is required as part of the Reinforcement Learning solution. Therefore, this survey’s aim is to assist in the development of new Text Game problem settings and solutions for Reinforcement Learning informed by natural language. Specifically, this survey: 1) introduces the challenges in Text Game Reinforcement Learning problems, 2) outlines the generation tools for rendering Text Games and the subsequent environments generated, and 3) compares the agent architectures currently applied to provide a systematic review of benchmark methodologies and opportunities for future researchers.


Introduction
given by Narasimhan et al. [2015] where the player takes the action 'Go East' to cross the bridge.
Reinforcement Learning (RL) has shown human-level performance in solving complex, single setting virtual environments Mnih et al. [2013] & Silver et al. [2016].However, applications and theory in RL problems have been far less developed and it has been posed that this is due to a wide divide between the empirical methodology associated with virtual environments in RL research and the challenges associated with reality Dulac-Arnold et al. [2019].Simply put, Text Games provide a safe and data efficient way to learn from environments that mimic language found in real-world scenarios Shridhar et al. [2020].
Natural language (NL) has been introduced as a solution to many of the challenges in RL Luketina et al. [2019], as NL can facilitate the transfer of abstract knowledge to downstream tasks.However, RL approaches on these language driven environments are still limited in their development and therefore a call has been made for an improvement on the evaluation settings where language is a first-class component.
Text Games gained wider acceptance as a testbed for NL research following work from Narasimhan et al. [2015] who leveraged the Deep Q Network (DQN) framework for policy learning on a set of synthetic textual games.Text Games are both partially observable (as shown in Figure 1) and include outcomes that make reward signals simple to define, making them a suitable problem for Reinforcement Learning to solve.However, research so far has been performed independently, with many authors generating their own environments to evaluate their proposed architectures.This lack of uniformity in the environments makes comparisons between authors challenging and a need for structuring recent work is essential for systematic comparability.
Formally, this survey provides the first systematic review of the challenges posed by the generation tools designed for Text Game evaluation.Furthermore, we also provide details on the generated environments and resultant RL agent architectures used to solved them.This acts as a complement to prior studies in the challenges of real-world RL and RL informed by NL by Dulac-Arnold et al. [2019] & Luketina et al. [2019] respectively.

Text Game Generation
Thus far, there are two main generation tools for applying agents to interactive fiction games: TextWorld and Jericho.
TextWorld Côté et al. [2018] is a logic engine to create new game worlds, populating them with objects and generating quests that define the goal states (and subsequent reward signals).It has since been used as the generation tool for Treasure Hunter Côté et al. [2018], Coin Collector Yuan et al. [2018], CookingWorld Trischler et al. [2019] and the QAit Dataset Yuan et al. [2019].
Jericho Hausknecht et al. [2019a] was more recently developed as a tool for supporting a set of human-made interactive fiction games that cover a range of genres.These include titles such as Zork and Hitchhiker's Guide to the Galaxy.Unsupported games can also be played through Jericho but will not have the point-based scoring system that defines the reward signals.Jericho has been as the generator tool for CALM Yao et al. [2020] and Jericho QA Ammanabrolu et al. [2020a].

Environment Model
Reinforcement Learning is a framework that enables agents to reason about sequential decision making problems as an optimization process Sutton and Barto [1998].Reinforcement Learning requires a problem to be formulated as a Markov Decision Process (MDP) defined by a tuple < S, A, T, R, γ > where S is the set of states, A is the set of actions, T is the transition probability function, R the reward signal and γ the discount factor.
Given an environment defined by an MDP, the goal of an agent is to determine a policy π(a|s) specifying the action to be taken in any state that maximizes the expected discounted cumulative return ∞ k=0 γ k r k+1 .In text games however, the environment states are never directly observed but rather textual feedback is provided after entering a command.As specified by Côté et al. [2018], a text game is a discrete-time Partially Observed Markov Decision Process (POMDP) defined by < S, A, T, Ω, O, R, γ > where we now have the addition of the set of observations (Ω) and a set of conditional observed probabilities O. Specifically, the function O selects from the environment state what information to show to the agent given the command entered to produce each observation o t ∈ Ω.

Handicaps
Before introducing the challenges of the environments, it is important to understand the possible limitations that can be imposed by the generation tools to reduce the complexity of each problem.These handicaps are typically used to limit the scope of the challenges being faced at any one time for more rigorous comparisons and for simplicity.
It has been noted that TextWorld's Côté et al. [2018] generative functionalities explicit advantage in that it can be used to focus on desired subset of challenges.For example, the size of the state space can be controlled and how many commands are required in order to reach the goal.Evaluation of specific generalisability measures can also be improved by controlling the training vs testing variations.
The partial observability of the state can also be controlled by augmenting the agent's observations.It is possible for the environment to provide all information about the current game state and therefore reduce the amount an agent must explore to determine the world, relationships and the objects contained within.
Furthermore, the complexity of the language itself can be reduced by restricting the agent's vocabulary to in-game words only or the verbs to only those understood by the parser.The grammar can be further simplified by replacing object names with symbolic tokens.It is even possible for the language generation to be avoided completely by converting every generated game into a choice-based game where actions at each timestep are defined by a list of pre-defined commands to choose from.
Rewards can simplified with more immediate rewards during training based on the environment state transitions and the known ground truth winning policy rather than simply a sparse reward at the end of a quest as normally provided.
Actions are defined by text commands of at least one word.The interpreter can accept any sequence of characters but will only recognize a tiny subset and moreso only a fraction of these will change the state of the world.The action space is therefore enormous and so two simplifying assumptions are made: -Word-level Commands are sequences of at most L words taken from a fixed vocabulary V.
-Syntax Commands have the following structure -verb[noun phrase [adverb phrase]] where [...] indicates that the sub-string is optional.
Jericho Hausknecht et al. [2019a] similarly has a set of possible simplifying steps to reduce the environment's complexity.Most notably, each environment provides agents with the set of valid actions in each game's state.It achieves this by executing a candidate action and looking for the resulting changes to the world-object-tree.To further reduce the difficulty of the games, optional handicaps can be used: • Fixed random seed to enforce determinism • Use of load, save functionality • Use of game-specific templates and vocabulary • Use of world object tree as an auxiliary state representation or method for detection player location and objects • Use of world-change-detection to identify valid actions Jericho also introduces a set of possible restrictions to the action-space.
• Template-based action spaces separate the problem of picking actions into two (i) picking a template (e.g."take ___ from ___"); (ii) filling the template with objects (e.g."apple", "fridge").Essentially this reduces the issue to verb selection and contextualised entity extraction.
• Parser-Based action spaces require agent to generate a command word-per-word, sometimes following a pre-specified structures similar to (verb, object_1, modifier, object_2).
• Choice-based requires agents to rank predefined set of actions without any option for "creativity" from the model itself.
Lastly, the observation space may be enchanced with the outputs of bonus commands such as "look" and "inventory".These are commands agent can issue on its own but are not considered an actual step in the exploration process that could be costly and produce risks in the real-world to ask for.

Challenges and Posed Solutions
Environments built from the TextWorld generative system bounds the complexity by the set number of objects available.For example, Côté et al. [2018] introduce 10 objects including the logic rules for doors, containers and keys, where complexity can be increased by introducing more objects and rules into the generation process.The most challenging environments are defined in Jericho as these contain 57 real Interactive Fiction games which have been designed by humans, for humans.Specifically, these environments contain more complexity in forms of stochasticity, unnatural interactions, unknown objectives -difficulties originally created to trick and hamper players.
The design and partially observed representation of text games creates a set of natural challenges related to reinforcement learning.Furthermore, a set of challenges specific to language understanding and noted by both Côté et al. [2018] & Hausknecht et al. [2019a] are specified in detail in this section.

Partial Observability
The main challenge for agents solving Textual Games is the environment's partial observability; when observed information is only representative of a small part of the underlying truth.The difference between the two is often unknown and can require extensive exploration and failures for an agent to understand the game's connections between what is observed and how this relates to its actions.
A related additional challenge is that of causality which is when an agent moves away from a state to the next without completing pre-requisites of future states.For example, an agent is required to use a lantern necessary to light its way but may have to backtrack to previous states if this has not been obtained yet.An operation which becomes more complex with time and increases as the length of the agent's trajectory increases.
Handcrafted reward functions have been proved to work for easier settings, like CoinCollector Yuan et al. [2018], but real-world text games can require more nuanced approaches.Go-Explore has been used to find high-reward trajectories and discover under-explored states Madotto et al. [2020], Ammanabrolu et al. [2020a] where more advanced states are given higher priority over states seen early on in the game by a weighted random exploration strategy.Ammanabrolu et al. [2020a] have expanded on this with a modular policy approach aimed at noticing bottleneck states with a combination of a patience parameter and intrinsic motivation for new knowledge.The agent would learn a chain of policies and backtrack to previous states upon getting stuck.
Heuristic-based approaches have been used by Hausknecht et al. [2019b] to restrict navigational commands to only after all other interactive commands have been exhausted.
Leveraging past information has been proven to improve model performance as it allows to limit the partial observability aspect of the games.Ammanabrolu and Hausknecht [2020] propose using a dynamically learned Knowledge Graph (KG) with a novel graph mask to only fill out templates with entities already in the learned KG.

Large State Space
Whilst Textual Games, like all RL problems, require some form of exploration to find better solutions, some papers focus specifically on countering that natural overfitting of RL by actively encouraging exploration to unobserved states in new environments.For example, Yuan et al. [2018] achieved this by setting a reward signal with a bonus for encountering a new state for the first time.This removes the agents capability for high-level contextual knowledge of the environment in favor of simply searching for unseen states.
Subsequent work by Côté et al. [2018] -Treasure Hunter -has expanded on this by increasing the complexity of the environment with additional obstacles such as locked doors that require colour matching keys requiring basic object affordance and generalisation ability.
In a similar vein to Treasure Hunter, where in the worst case agents have to traverse all states to achieve the objective, the location and existence settings of QAit Yuan et al. [2019] require the same with addition of stating the location or existence of an object in the generated game.
These solutions are also related to the challenge of Exploration vs Exploitation that is commonly referenced in all RL literature Sutton and Barto [1998].

Large, Combinatorial and Sparse Action Spaces
Without any restrictions on length or semantics, RL agents aiming to solve games in this domain face the problem of an unbounded action space.Early works limited the action phrases to two words sentences for a verb-object, more recently combinatory action spaces are considered that include action phrases with multiple verbs and objects.A commonly used method for handling combinatory action spaces has been to limit the agent to picking a template T and then filling in the blanks with entity extraction Elimination Network for approximation of the admissibility function: whether the action taken changes the underlying game state or not.The second has been used in limited scope with pointer softmax models generating commands over a fixed vocabulary and the recent textual observation.The CALM generative model, leveraging a fine-tuned GPT-2 for textual games, has proved to be competitive against models using valid action handicaps.

Long-Term Credit Assignment
Assigning rewards to actions can be difficult in situations when the reward signals that are sparse.Specifically, positive rewards might only obtained at the successful completion of the game.However, environments where an agent is unlikely to reach the positive end game through random exploration, provide rewards for specific subtasks such as in Murugesan et al. [2020a], Trischler et al. [2019], Hausknecht et al. [2019a].The reward signal structured in this way also aligns with hierarchical approaches such as in Adolphs and Hofmann [2020].Lastly, to overcome the challenges presented with reward-sparsity, various hand-crafted reward signals have been experimented with Yuan et al. [2018], Ammanabrolu et al. [2020a].

Understanding Parser Feedback & Language Acquisition
LIGHT Urbanek et al. [2019], a crowdsourced platform for the experimentation of grounded dialogue in fantasy settings that differs from the previous action-oriented environments by requiring dialogue with humans, embodied agents and the world itself as part of the quest completion.The authors design LIGHT to investigate how 'a model can both speak and act grounded in perception of its environment and dialogue from other speakers'.Ammanabrolu et al. [2020b] extended this by providing a system that incorporates '1) large-scale language modelling based commonsense reasoning pre-training to imbue the agent with relevant priors and 2) a factorized action space of commands and dialogue'.Furthermore, evaluation can be performed against a dataset collected of held-out human demonstrations.

Commonsense Reasoning & Affordance Extraction
As part of their semantic interpretation, Textual Games require some form of commonsense knowledge to be solved.For example, modeling the association between actions and associated objects (opening doors instead of cutting them, or the fact that taking items allows to use them later on in the game).Various environments have been proposed for testing procedural knowledge in more distinct domains and to asses the agent's generalisation abilities.
For example, Trischler et al. [2019] proposed the 'First TextWorld Problems' competition with the intention of setting a challenge requiring more planning and memory than previous benchmarks.To achieve this, the competition featured 'thousands of unique game instances generated using the TextWorld framework to share the same overarching themean agent is hungry in a house and has a goal of cooking a meal from gathered ingredients'.The agents therefore face a task that is more hierarchical in nature as cooking requires abstract instructions that entail a sequence of high-level actions on objects that are solved as sub-problems.
Furthermore, TW-Commonsense Murugesan et al. [2020a] is explicitly built around agents leveraging prior commonsense knowledge for object-affordance and detection of out of place objects.
Two pre-training datasets have been proposed that form the evaluation goal for specialised modules of RL agents.
The ClubFloyd dataset1 provides human playthroughs of 590 different text based games, allowing to build priors and pre-train generative action generators.Likewise, the Jericho-QA Ammanabrolu et al. [2020a] dataset provides context at a specific timestep in various classical IF games supported by Jericho, and a list of questions, enabling pre-training of QA systems in the domain of Textual Games.They also used the dataset for fine-tuning a pre-trained LM for building a question-answering-based Knowledge Graph.
Lastly, ALFWorld Shridhar et al. [2020] offers a new dimension by enabling the learning of a general policy in a textual environment and then test and enhance it in an embodied environment with a common latent structure.The general tasks also require commonsense reasoning for the agent to make connections between items and attributes, e.g.(sink, "clean"), (lamp, "light").Likewise, the attribute setting of QAit Yuan et al. [2019] environment demands agents to understand attribute affordances (cuttable, edible, cookable) to find and alter (cut, eat, cook) objects in the environment.

Knowledge Representation
It can be specified that at any given time step, the game's state can be represented as a graph that captures observation entities (player, objects, locations, etc) as vertices and the relationship between them as edges.As Text Games are partially observable, an agent can track its belief of the environment into a knowledge graph as it discovers it, eventually converging to an accurate representation of the entire game state Ammanabrolu and Riedl [2019].
In contrast to methods which encode their entire KG into a single vector (as shown in Ammanabrolu and Hausknecht [2020], Ammanabrolu et al. [2020a]) Xu et al. [2020] suggest an intuitive approach of using multiple sub-graphs with different semantical meanings for multi-step reasoning.Previous approaches have relied on predefined rules and Stanford's Open Information Extraction Angeli et al. [2015] for deriving information from observations for KG construction, Adhikari et al. [2020] have instead built an agent, which is capable of building and updating it's belief graph without supervision.
Reframing the domain of Textual-Games, Guo et al.
[2020] consider observations as passages in a multi-passage RC task.They use an object-centric past-observation retrieval to enhance current state representations with relevant past information and then apply attention to draw focus on correlations for action-value prediction.

Benchmark Environments and Agents
Thus far, the majority of researchers have independently generated their own environments with TextWorld Côté et al. [2018].As the field moves towards more uniformity in evaluation, a clear overview of which environments are already generated, their design goals and the benchmark approach is needed.
Table 1 shows the publicly available environments and datasets.Much of the recent research has been published within the past two years (2018-2020).Jericho's Suite of Games (SoG) Hausknecht et al. [2019a] is a collection of 52 games and therefore has its results averaged across all games included in evaluation.
We find that many of the environments focus on exploring increasingly complex environments to 'collect' an item of some kind ranging from a simple coin to household objects.This is due to the TextWorld generator well defined logic rules for such objects but can also be a limitation on the possible scope of the environment.
When evaluating an agent's performance, three types of evaluation settings are typically considered: • Single Game evaluate agents in the same game under the same conditions, • Joint settings evaluate agents trained the same set of games that typically share some similarities in the states seen, • Zero-Shot settings evaluate agents on games completely unseen in training.
The difficulty settings of the environments are defined by the complexity of the challenges that the agents are required to overcome.In most cases, these have been limited to just a few levels.However, CookingWorld defines the challenge by a set of key variables that can be changed seperately or in combination and therefore does not offer clear discrete difficulty settings.
The max number of rooms & objects depends heavily on the type of task.Coin Collecter for example has only 1 object to find as the complexity comes from traversing a number of rooms.Alternatively, CookingWorld has a limited number of rooms and instead focuses on a large number of objects to consider.Zork is naturally the most complex in this regards as an adaptation of a game designed for human players.Likewise, the complexity of the vocabulary depends heavily on the task but is clear to see that the environments limit the number of Action-Verbs for simplicity.length of optimal trajectory.

Agent Architectures
A Deep-RL agent's architecture (see Figure 2) consists of two core components (i) state encoder and (ii) and action scorer Mnih et al. [2013].The first is used to encode game information such as observations, game feedback and KG representations into state approximations.The encoded information is then used by an action selection agent to estimate the value of actions in each state.
Table 2 provides an overview of recent architectural trends for comparison.We find that the initial papers in 2015 used the standard approaches of LSTM or Bag of Words for the encoder and a Deep Q-Network (DQN) for the action selector.More recent developments have been experimenting with both parts towards improved results.Notably, a range of approaches for the encoding have been introduced with Gated Recurrent Unit's (GRUs) have become the most common in 2020.Whereas, there have been fewer variations in the choice of action selector where either Actor-Critic (A2C) or a DQN is typically used.Furthermore, the use of Knowledge Graphs and Pre-trained Tranformers is limited and many of the works that use these were published in 2020.
Most of the agents were applied to either TextWorld/Cookingworld or Jericho.We typically find that alternative environments align to consistent setups; either the authors create a new game that mimics simple real-world rules (similar to TextWorld) or they apply their methods to well know pre-existing games (similar to Jericho).Specifically, Narasimhan et al. [2015] used a self generated two games themselves: 'Home World' to mimic the environment of a typical house and 'Fantasy World' that is more challenging and akin to a role-playing game.Alternatively, He et al.
[2016] used a deterministic text game 'Saving John' and and a larger-scale stochastic text game 'Machine of Death', both pre-existing from a public libary.Encoders used include both simplistic state encodings in the form of Bag of Words (BoW), but also recurrent modules like: Gated Recurrent Unit (GRU) Cho et al.Task Specific Pre-Training entails heuristically establishing a setting in which a submodule learns priors before interacting with training data.For example, Adolphs and Hofmann [2020] pretrain on a collection of food items to improve generalizability to unseen objects and Adhikari et al. [2020] pretrain a KG constructor on trajectories of similar games the agent is trained on.Chaudhury et al. [2020] showed that training an agent on pruned observation space, where the semantically least relevant tokens are removed in each episode to improve generalizability to unseen domains whilst also improving the sample efficieny due to requiring less training games.Furthermore, Jain et al. [2020] propose learning different action-value functions for all possible scores in a game, thus effectively learning separate value functions for each subtask of a whole.
Lastly, the action-space typically varies between template or choice based depending on the type of task.Only two papers have considered a parser based approach: Narasimhan et al. [2015] and Madotto et al. [2020].

Benchmark Results
The following section summarises some of the results published thus far as a means to review the performance of the baselines as they are presented by the original papers to be used for future comparisons.
Treasure Hunter was introduced by Côté et al. [2018] as part of the TextWorld formalisation and inspired by a classic problem to navigate a maze to find a specific object.The agent and objects are placed in a randomly generated map.A coloured object near the agent's start location provides an indicator of which object to obtain provided in the welcome message.A straight forward reward signal is defined as positive for obtaining the correct object and negative for an incorrect object with a limited number of turns available.
Increasing difficulties are defined by the number of rooms, quest length and number of locked doors and containers.Levels 1 to 10 have only 5 rooms, no doors and an increasing quest length from 1 to 5. Levels 11 to 20 have 10 rooms, include doors and containers that may need to be opened and quest length increasing from 2 to 10. Lastly, levels 21 to 30 have 20 rooms, locked doors and containers that may need to be unlocked and opened and quest length increasing from 3 to 20.
For the evaluation, two state-of-the-art agents (BYU Fulda et al. [2017] and Golovin Kostka et al. [2017]) were compared to a choice-based random agent in a zero-shot evaluation setting.For completeness, each was applied to 100 generated games at varying difficulty levels up to a maximum of 1,000 steps.The results of this are shown in table 3 where we note that the random agent performs best but this is due to the choice-based method removing the complexity of the compositional properties of language.The authors noted that may not be directly comparable to the other agents but provides an indication of the difficulty of the tasks.4. However there are some variances in the split of training and test games, making it challenging to compare the results.Most models were trained using the games split of 4,400/514 specified before, but there are variances as some split it as 3,960/440 and some do not state the split at all.Table 4 shows the results of this comparison with a percentage score relative to the maximum reward.Recent research developments for interactive fiction games have generated a set of environments and tested their architectures for generalization and overfitting capabilities.With the comparisons in this work and a uniform set of environments and baselines, new architectures can be systematically contrasted.Currently, most methodologies focus on either an agent's ability to learn efficiently or generalise knowledge.A continued development in this direction is the primary motive for much of the recent research and could lead to solutions that work sufficiently well for the primary real-world challenge of limited and expensive data.

Random
The most recent advances the agent's performance to reach a goal given the linguistic challenge have come from utilising pre-trained transformer models.This has been supported by work published in the NLP community with methods such as BERT Yin et al. [2020], GPT-2 Yao et al. [2020] and ALBERTAmmanabrolu et al. [2020a] and continued developments from this community will support the advancements of future agents architectures.
Similarly, the use of dynamically learned knowledge graphs have become common in recent works and further developments from the NLP community (such as Das et al. [2019]) could enable the use of pre-training knowledge graphs on readily available text corpora before training on the environment itself.
However, there are no methods that currently consider interpretability with language as a primary goal.This is not a challenge unique to Text Games or Reinforcement Learning as calls for research have been made in similar machine learning domains Chu et al. [2020].A notable methodology that is worth considering is the Local Interpretable Model-Agnostic Explanations (LIME) introduced by Ribeiro et al. [2016].
Furthermore, as noted by Luketina et al. [2019], grounding language into the environments is still an open problem that has yet to be addressed fully.Urbanek et al. [2019] introduces a Text Game environment specifically for grounding dialogue as a first step but this is still an open problem.
Lastly, from the contributions analysed in this survey only 5 papers report the amount of time and resources their methods needed for training.Continuing the trend of publishing these specifications is essential in making results more reproducible and applicable in applied settings.Reducing the amount of resources and training time required as a primary motive allows for the domain to be practical as a solution and also accessible given that not all problems allow for unlimited training samples on high performance machines.

Figure 1 :
Figure 1: Sample gameplay from a fantasy Text Game as given by Narasimhan et al.[2015]  where the player takes the action 'Go East' to cross the bridge.

Figure 2 :
Figure 2: Overview of the Architecture Structure of Agents Applied to a Simple Text Game Example.

Table 1 :
Environments for Textual-Games; Eval: (S)Single, (J) Joint, (ZS) Zero-Shot; GEN: engine support for generation of new games, #Diffic.: number of difficulty settings, #rooms & #objects: number of rooms and objects per game, AVS: size of Action-Verb Space (NS=Not Specified), len(ot): mean number of tokens in the observation ot, |V |: Vocabulary size and, |quest|:

Table 3 :
Trischler et al. [2019]ed to Treasure Hunter's one-life tasks.CookingWorld is the second well-established environment developed using TextWorld byTrischler et al. [2019], is used in joint and zero-shot settings, which both enable testing for generalization.The former entails training and evaluating the agent on the same set of games, while the latter uses an unseen test set upon evaluation.The fixed dataset provides 4, 400 training, 222 validation and 514 test games with 222 different types of games in various game difficulty settings and the current best results are stated in Table

Table 4 :
Results on the public CookingWorld dataset; OoD -out of domain Jericho's Suite of Games was defined byHausknecht et al. [2019a]and has been used for testing agent's in a single game setting.As the suite provides 57 games, of which a variation of 32 are used due to excessive complexity, then the agents are initialized, trained and evaluated on each of them separately.Agent's are assessed over the final score, which is by convention averaged over the last 100 episodes (which in turn is usually averaged over 5 agents with different initialization seeds).The current state of the art results on the Jericho gameset can be seen in Table5where numeric results are shown relative to the Maximum Reward with the aggregated average percentage shown in the final row 2 .

Table 5 :
Blue: likely to be solved in near future.Orange: progress is likely, significant scientific progress necessary to solve.Red: very difficult even for humans, unthinkable for current RL.MaxR -Maximum possible reward ;|T | -number of templates; |V | -size of vocabulary set 4 Conclusion and Future Work