A Systematic Literature Review of the Successors of “NeuroEvolution of Augmenting Topologies”

NeuroEvolution (NE) is a learning method that uses Evolutionary Algorithms (EA) to optimize the parameters of Artificial Neural Networks (ANNs) (Stanley et al., 2019). In early neuroevolutionary methods, evolutionary optimization methods (e.g., Genetic Algorithms (GAs)) were used for learning the connection weights of fixed topology ANNs (Whitley et al., 1990; Montana and Davis, 1989; Gomez and Miikkulainen, 1997; Moriarty and Miikkulainen, 1996; Gomez and Miikkulainen, 1998). However, since the definition of a network's topology has a major effect on its performance, new methods appeared that optimize both the weights and the topology (Maniezzo, 1994; Liu and Yao, 1996; Yao and Liu, 1998; Moriarty and Miikkulainen, 1995; Stanley and Miikkulainen, 2002b). These methods are known as Topological and Weight Evolving Artificial Neural Networks (TWEANNs).

TWEANNs offer significant advantages over fixed topology ANNs, as finding the optimal topology of an ANN requires time-consuming evaluations of potential architectures. Especially, in complex problems, the number of neurons and connections that are required scales with the complexity of the problem (Manning and Walsh, 2012) and thus the manual definition of the optimal topology is even more difficult. Moreover, the topology defines the size of the search space: selecting a fixed topology smaller than the optimal means that the search is performed in a lower dimensional space and thus the optimal solution will not be found. On the other hand, selecting a bigger topology than the optimal one implies that the search is performed in an unnecessarily high dimensional space. In TWEANNs the problem of identifying the right ANN topology is tackled by their ability to automatically discover the optimal architecture.

In 2002, a revolutionary TWEANN method called NeuroEvolution of Augmenting Topologies (NEAT) was invented by Kenneth Stanley and Risto Miikulainen (Stanley and Miikkulainen, 2002b). The method provides solutions to problems encountered in NE by facilitating the crossover between individuals of different length, evolving networks by adding new structure when necessary, and protecting structural innovations by organizing them in species.

Today, 18 years later, the NEAT algorithm is still relevant as the large number of published papers extending NEAT or applying NEAT-based methods on challenging problems shows. To our knowledge, no systematic literature review (SLR) of NEAT-based successors exists. Literature review papers that present an overview of NE methods in general can be found (Yao, 1993, 1999; Floreano et al., 2008; Ding et al., 2013; Risi and Togelius, 2015; D'Ambrosio et al., 2014; Stanley et al., 2019). However, none of them uses a systematic review protocol. Instead, they report on the most prominent and important papers in the field. The reviews (Yao, 1993, 1999; Floreano et al., 2008) are very detailed papers that cover the field of NE until 2008. More recent review papers (e.g., Ding et al., 2013; Risi and Togelius, 2015) exist, but their scope is different than the current article, as Ding et al. (2013) describe basic theories and methods in NE and Risi and Togelius (2015) discuss NE's applications in games. A chapter in Growing Adaptive Machines by D'Ambrosio et al. (2014) gives an overview of the methods that followed HyperNEAT within five years. Finally, the most recent paper (Stanley et al., 2019) gives an overview of different aspects of recent NE techniques and discusses their potential for application in the field of deep learning.

In the current article, we present a new categorization of the NEAT successors based on criteria defined in this article. This article can serve as a guidance manual that allows researchers to propose a new NEAT-based method by starting from the most appropriate baseline method or the most recent method or to select the method that is more appropriate to solve their problem.

Toward this purpose, detailed tables presenting a fine-grained categorization of the methods according to three main criteria, as well as an illustration (see Figure 1 on p. 13) of the trajectory of the advances in NEAT-based methods are provided. With this new categorization scheme researchers can better position their work in the landscape of existing methods, or select an existing method that fits best the properties of their problem's landscape/search space, or that evolves ANNs with particular properties or that combines principles from different research fields. Moreover, researchers who are interested in benchmarking their method to the state of the art can consult the detailed tables in the Appendix regarding the employed datasets and performance metrics. All the NEAT successors identified by the review protocol are described in detail in the Appendix of this article. Finally, we discuss the findings and the shortcomings of existing approaches, the popularity of NEAT-based algorithms nowadays, and their future relevance in the era of deep learning.

The remainder of this article is organized as follows. Section 2 describes the methodology engaged for conducting the SLR and Section 3 presents the answers to the main research questions regarding the NEAT method. An extended list with detailed answers to all the research questions for all the identified methods can be found in the Appendix. Section 4 presents the proposed categorization clusters and detailed tables with the findings of the review protocol. Finally, Sections 5 and 6 contain the discussion and conclusion parts of this article.

With this review article we want to present an objective historical account of NEAT successors. Toward this purpose, we raise the following question: “which methods are based on and succeeded NEAT within a period of 15 years?” A systematic literature review is a type of review that satisfies this objective (Bettany-Saltikov, 2010) as it allows for the findings to be presented in an objective and independent summary (Hemmingway and Brereton, 2009). Since a systematic review article is characterized by the development and use of a review protocol, we followed published guidelines (Keele, 2007; Kofod-Petersen, 2012) on “how to conduct a systematic literature review paper.” For a comprehensive review on modern neuroevolution, not necessarily based on NEAT methods, but focusing on current trends, readers are advised to read a paper by Stanley et al. (2019).

Conducting a rigorous and repeatable SLR requires a review protocol (Bettany-Saltikov, 2010). We followed the guidelines according to Keele (2007) and Kofod-Petersen (2012). First, we defined the objective and the research questions that this review was going to address. Secondly, we designed the search strategy toward answering the research questions by defining the search terms and the inclusion and exclusion criteria for selecting the relevant literature. After a pilot study we refined both the search terms and the selection criteria. In the following phase, we created a Quality Assessment check-list to evaluate the selected literature. Finally, we determined a data extraction form inspired by the one in Wen et al. (2012) for documenting the main features of each method.

The objective of this SLR is to summarize, categorize, and clarify NEAT-based NE methods. These methods will be referred to as x-NEAT.¹ Toward this objective, we raised the following Research Questions (RQ):

1. RQ1: What are the main principles and features of each method? (Principles)

2. RQ2: How are the ANNs encoded into the genome? (Encoding) The purpose is to identify the different encoding schemes that are used for representing the ANNs phenotypes in the GAs' genome.

3. RQ3: How does the x-NEAT method perform compared to others? Which performance metrics are used to make this comparison? (Performance)

In order to identify the relevant papers succeeding NEAT (Stanley and Miikkulainen, 2002b), we had to define the search terms by identifying the major terms and alternative spellings and synonyms. We used the Boolean OR to combine synonyms and alternative spellings and the Boolean AND to link the major terms. The search was performed on the 18 April 2018 and we chose to include all the papers published until the end of 2017. After refinement, the resulting search term is the following ((neuro?evolution OR neuroevolution* OR evolv* neural networks) AND (*neat OR augment* topologies)) AND (EXCLUDE (PUBYEAR, 2018)).

With the search term, we searched for published journal and conference papers in the electronic databases of Web of Science (WoS) (all databases) and Scopus. The search resulted in 225 publications from all the databases of WoS and 121 publications from Scopus. We searched each database separately and gathered the papers together. Excluding the common papers between the two databases resulted in 232 publications which had to be evaluated on their relevance and quality.

To select the relevant papers we read all the abstracts and we applied the inclusion (IC) and exclusion (EC) criteria defined in Table 1. Forty papers were selected based on IC1, 25 based on IC2, and 5 based on both IC1 and IC2. Sixty-two papers were excluded because of EC1, 7 because of EC2, 42 based on EC3, 39 based on EC4, 1 based on EC1 and EC3, and 1 based on EC2 and EC3.

The quality of each relevant paper was evaluated according to the Quality Assessment Criteria (QAC) defined in the format of questions and presented in Table 2. If a question was addressed by the paper, it could take one of the answers: “yes,” “partly,” and “no,” which received one, one-half, and zero points, respectively. If the question was not addressed at all by the paper, then it received the answer “non-applicable” excluding it from the calculation of the final result. In this way, a paper's final score was calculated by averaging the score of each question. Nine papers had a score $∈[0,0.5]$ and they were excluded from the study.

Application of the IC, EC, and the QAC resulted in 61 papers whose acronyms and obtained scores are presented in Table 4. From these, we collected the necessary data to answer the RQs. To keep the most important information from each paper, we created cards as shown in Table 3 similar to Wen et al. (2012).

NeuroEvolution of Augmenting Topologies (NEAT) (Stanley and Miikkulainen, 2002b) is a TWEANN method that enables the learning of the structure of ANNs at the same time it optimizes their connectivity weights. When NEAT was proposed in 2002, it provided solutions to important research questions of that time. The proficiency of the method is attributed to the three main innovations for which ablation studies showed that each of the introduced components is crucial to the NEAT's performance. The three main features of NEAT are described in Section 3.1.

Historical Markings to Deal with the Competing Conventions Problem.  One crucial issue in NE is the Competing Conventions Problem, also known as the Permutation Problem (Radcliffe, 1993) that appears when there is more than one way to represent an ANN and because structures evolve independently in different networks. In particular, the problem occurs during the crossover of two genomes that are encoded differently even though they represent the same solution. As a consequence, the resulting offspring might suffer from loss of functionality. To overcome this problem, the system should be able to identify identical structures. NEAT deals with this issue by means of historical markings, that is, by assigning an increasing innovation number when a new gene is added to the genome. A new gene is introduced in the system by structural mutations that add a new connection or a new node in the network. The historical markings act as chronological indicators that facilitate crossover by identifying homologous sections between different networks. The innovation number of each gene is inherited by the offspring, facilitating the retaining of its historical origin throughout evolution.

Biasing the Search toward Smaller Structures.  NEAT starts the evolution with a uniform population of minimal structures, that is, fully connected networks with no hidden nodes and it evolves more complex networks (complexification) by introducing new nodes and connections through structural mutations. These topological innovations are maintained only if they are found to be useful after fitness evaluation, that is, if they can increase the network's performance. In this way, NEAT tends to evolve smaller structures.

NEAT uses direct encoding to encode the phenotypes (ANNs) in the genotype. In direct genetic encoding, there is a one-to-one mapping between the networks' phenotypes and genotypes. One genome is used for encoding the nodes and a second genome for encoding the connections. The networks' nodes are encoded with a set of node genes while the connections among the nodes are encoded with a set of connection genes. The node genes have fields that indicate the type of the node, that is, whether the node is an input, an output, or a hidden node, whereas the connection genes indicate the connections established between the nodes in the node genome. Each connection gene consists of 5 fields: the id of the node receiving the connection (in-node), the id of the node from where the connection begins (out-node), the weight of the connection, an enabled bit that specifies whether the connection is enabled or not, and an innovation number which allows finding corresponding genes during mating.

NEAT is tested on the XOR problem and its performance is evaluated with the average number of generations that are required to solve it. Moreover, NEAT is compared to four other NE systems on the double pole balancing problem (a benchmark Reinforcement Learning (RL) task), in terms of number of generations, their ability to generalize and the size of the evolved networks. NEAT converges faster, but its ability to generalize is not statistically different from other systems.

In this section, we give a summary of the findings of the review protocol. The 61 methods that fulfill the inclusion criteria and pass the quality assessment threshold are presented in Table 4 along with the obtained scores. Detailed answers to the three research questions for all the methods are presented in the Appendix.

To begin with, in Table 5, we present a categorization of NEAT's successors based on the employed encoding scheme, that is, the way the mapping between GA's genotype and ANN's phenotype happens. We identify three types of encoding: direct, indirect, and hybrid.

Direct encoding refers to a one-to-one mapping between the genotype and the phenotype, such as NEAT's node and connection genes described in Section 3.2. On the other hand, in indirect encoding a set of rules is used to map the genotype to the phenotype. Compositional Pattern Producing Networks (CPPNs) (Stanley, 2007), details presented in the Appendix, are an example of indirect encoding (Stanley et al., 2019). CPPNs, which behave as ANNs, can be evolved by NEAT to output a spatial pattern in the hyperspace which corresponds to a connectivity pattern in a substrate of nodes. Hybrid encoding refers to an alternation between direct and indirect encoding.

Analyzing how the ANNs are encoded in the genotype, we found that approximately two-thirds of the methods use direct encoding, while one method proposes a hybrid encoding. Most of the methods with direct encoding use NEAT's encoding without modifications. However, there are methods that modify it by adding new fields according to the characteristics of the proposed method. The methods using indirect encoding either propose a new type of encoding or use HyperNEAT's encoding. When necessary, modifications on HyperNEAT's encoding are made to meet the requirements of the proposed method, for example, by changing the initial topology of the CPPNs to include more input, output, or hidden nodes.

Although NE methods can be traditionally categorized based on the encoding used for mapping the genotype to the phenotype, as presented in Section 4.1, in this article we perform a more fine-grained classification. Based on three criteria presented next, the NE methods are classified into three clusters, each of which consists of a number of subclusters that result in 18 subclusters in total. We started the categorization from subclusters that considered research questions we wanted to address and then we regrouped them in a higher level of clusters to increase the readability and get a more structured cluster description.

• Cluster 1: methods that consider issues relevant to the search space or the fitness landscape. This cluster is about properties of the search space and the fitness landscape and not about descriptive features of a certain method.

• Cluster 2: hybrid methods, that is, methods that employ principles from NE and another field of EC or Machine Learning (ML).

• Cluster 3: methods that evolve ANNs with particular properties.

NE methods are classified into the following subclusters based on which search space/landscape issues they solve, which include:

• a problem/search space with multiple objectives. A multiobjective optimization task is a domain where the simultaneous optimization of multiple conflicting objectives is required (Branke et al., 2008).

• a search space characterized by many irrelevant or redundant features. A feature is a measurable property or characteristic that is used to describe a task (Bishop, 2006). In this way, the instances of a problem are represented as multidimensional points in an n-dimensional space (where n is the number of features describing the problem), which is difficult to search.

• a deceptive landscape. A landscape is deceptive when the population of the GA is misguided away from the objective (Horn and Goldberg, 1995). In this case, a fitness function deceives the search by pointing the wrong way.

• a landscape characterized by uncertainty. An uncertain landscape is one where the fitness functions' optimum changes rapidly between generations (Krčah, 2012).

• a search space of an open-ended problem, i.e. a problem whose final solution is not finite (Stanley and Miikkulainen, 2004) and for which more complex (Maley, 1999) and novel (Standish, 2003) solutions are continuously generated.

• a space where evolution takes place online or in real time. In online evolution an ANN is evolved spontaneously without having prior information of the whole environment or task. This is in contrast with offline evolution when all the information is available from the beginning and the ANN is evolved before its final application to the target task. In real-time evolution, a constraint of the time for generating a solution exists. According to Cardamone et al. (2010), online evolution focuses on maximizing an agent's performance during the whole learning process, whereas real-time evolution aims to find a group of agents that perform well as fast as possible.

These methods combine NE principles with methods from the following fields:

• Evolutionary Computation (EC)

• Backpropagation (BP)

• Reinforcement Learning (RL)

• Unsupervised Learning (UL)

NE methods that can evolve ANNs with particular characteristics belong to this cluster. These properties include:

• modularity. A network is modular if it contains “highly connected clusters of nodes that are sparsely connected to nodes in other clusters” (Clune et al., 2013). A modular network consists of independent functional structures that can be separately optimized (Kashtan and Alon, 2005), can operate on separate inputs to perform a sub-task and are organized by an intermediary to produce the network's output (Azam, 2000).

• plasticity. An ANN is plastic when its connections' weights do not remain static during their lifetime but can change in response to the changing activation levels in the neurons they connect (Risi and Stanley, 2010).

• transfer learning ability, that is, transferring knowledge that is learned on one task to another one (Bahçeci and Miikkulainen, 2008).

• automatic ANN substrate configuration. This refers to methods that evolve not only the weights of a large scale ANN substrate, but also its topography, that is, the density and placement of neurons (Risi and Stanley, 2012a). Substrate is a specific terminology introduced in HyperNEAT (Stanley et al., 2009). For a detailed description please see Section A.13 in the Appendix.

• different types of nodes, that is, ANNs with nodes that are different than the sigmoid neuron units.

• large scale topologies, evolving large ANNs that define a high-dimensional search space. In this category we also include ANNs with a large number of input and output nodes to solve problems with high-dimensional input/output space.

• deep architectures. We refer to methods optimizing parameters of deep ANNs (DNNs).

• memory capacity. Augmenting an ANN with memory is essential for sequential problems requiring long-term memory.

The classification of the x-NEAT methods found by the review protocol is presented in Table 6; only two methods, TMO-NEAT (Marzullo et al., 2017) and Cascade-NEAT (Kohl and Miikkulainen, 2009), could not be categorized into any of the proposed clusters. It should be noted that this table is not an exhaustive list of all the methods in the field, just the ones that are returned by the inclusion and selection criteria of the protocol. The table is preferably read column-wise: a single tick in a column does not at all mean that the indicated paper is the only paper addressing the category. In the Appendix, we present detailed answers of the three RQs (principles, encoding, and evaluation) for each method. Figure 1 presents the trajectory of the x-NEAT methods' evolution. This is a graphical illustration of how x-NEAT methods emerged from the original NEAT method, that is, which method is being extended by another.

Table 6:

Proposed categorization of the x-NEAT methods.

View Large

Tables 8, 9, and 10 in the Appendix present the domains/datasets/tasks on which the x-NEAT methods of this review article are tested and the used performance metrics. Each dataset has one or more references that correspond to the x-NEAT method that employs it. In some cases, the original source of the dataset was available and this is included as the last citation in the list of provided citations. These tables can help researchers to pick up the method that is most suitable for their problem and find its detailed description in the Appendix. Also, it facilitates comparing and benchmarking new or existing NE methods by choosing the dataset and the performance metric used in the state of the art.

After the invention of NEAT, many methods have appeared that extend its functionality in various ways. We believe that the reason for this expansion is because NEAT is an easily understandable algorithm that works well on difficult problems and many researchers made its implementation publicly available. Moreover, we attribute this expansion to the fundamental importance of the NEAT principles, detailed in Section 3. NEAT incorporates specific biological principles (Stanley and Miikkulainen, 2002b) that have contributed to its success. The evolutionary process in NEAT resembles the process of natural evolution, as the evolved networks become more complex during their optimization. As in NEAT, the genome in nature does not have a fixed length, but new genes have been added during evolution through a process known as gene amplification (Darnell and Doolittle, 1986; Watson, 2004). Moreover, the solution that NEAT provides to the competing conventions problem is also inspired by natural evolution and in particular by the synapsis process, that is, the process of lining up homologous genes before crossover (Radding, 1982; Sigal and Alberts, 1972). Similarly, the speciation mechanism is inspired from nature's speciation mechanism that groups together individuals that share a common characteristic and implicitly protects innovation since the different species compete within their own niche (Stanley and Miikkulainen, 2002b). Although speciation as a notion was known in GAs, it was NEAT which introduced it in the TWEANNs, since NEAT offered a solution to the competing conventions problem with the use of historical markings. Before this solution, the definition of a compatibility metric, necessary for assigning individuals into species, was difficult to define. As far as the notion of evolving minimal architectures is concerned, before NEAT there were already attempts to control the size of the evolved topologies by including penalties in the fitness function. However, these attempts had the disadvantages of having to predefine the penalty and risking to have a different performance from the original non-penalized fitness function (Stanley and Miikkulainen, 2002b). By keeping the topologies minimal throughout evolution, NEAT tends to minimize the search space and to speed up the learning process (Stanley and Miikkulainen, 2002b). Before NEAT, starting with a uniform minimal population was not possible. Instead, TWEANNs started the evolution with topologically diversified networks because without speciation topological innovations could not survive (Stanley and Miikkulainen, 2002b). On the other hand, speciation in NEAT enabled the evolution to start minimally and diversify along evolution, since topological innovations would be protected. In conclusion, NEAT tackled problems known in the community in a unified and fundamentally strong manner.

A clear indication of how fundamentally important and globally applicable the NEAT principles are, can be found in the existence of a large number of papers whose method is inspired by the NEAT principles in, for example, NE with ontogeny (Inden, 2008), Genetic Programming (Drchal and Šnorek, 2013; Drchal and Snorek, 2012; Trujillo et al., 2016), and molecular programming (Dinh et al., 2014). Neat-GP (Trujillo et al., 2016) is an example of a method that adopts NEAT's principles (minimal topology initialization, speciation with fitness sharing and crossover by identifying the matching topologies) for evolving programs in Genetic Programming (GP) for regression and classification tasks. Neat-GP was able to achieve or improve the test fitness performance of standard GP while evolving smaller programs and reducing the computational cost and it shows that NEAT principles are fundamental and can be extended beyond neural networks.

From the tables provided in the Appendix of this review article as well as from Table 7, we can conclude that comparing the performance of different x-NEAT methods is difficult, as even methods belonging to the same subcluster are not all evaluated on the same datasets and do not employ the same performance metrics.

Different Performance Metrics.  The majority of the methods report on the algorithm's performance using the obtained fitness value. Similar metrics include the accuracy, the average error and the absolute error. Most of the methods obtain the mean value of these metrics calculated over the different algorithmic runs. Another metric is the ratio of runs when the algorithm was successful in solving the problem. Also, the majority of the methods report metrics to describe the algorithm's efficiency such as the number of generations and the computational time. Metrics describing the size of the evolved networks are also very common, for example, the number of evolved nodes and connections. Some methods are evaluated based on qualitative metrics, i.e. based on the user's interpretation of the evolved patterns and the agents' evolved behavior or morphology. Other papers evaluate the generalization ability of the proposed algorithm by testing the best ANN on a different task with different initial conditions. Finally, metrics to evaluate a specific characteristic of a method are proposed, such as the ratio between relevant and irrelevant features or the sum of absolute weights for methods performing feature selection, the number of agents that are dominated for coevolutionary methods and others.

Performance Comparisons.  Even though a newly proposed method that brings new insights in the field has a great value in its own, we observed that many research papers also compare their methods to other ones using a wide set of datasets and performance metrics. We believe that in order to facilitate comparisons, a good practice would be to report performance on a minimal set of universal performance metrics. Regarding the algorithms' performance, the accuracy, precision, recall, and confusion matrix for classification tasks and the accumulated reward for identical reinforcement learning tasks should be reported and be accompanied by statistical tests to indicate whether differences in performance are significant or not. In the papers reviewed in this study, a statistical hypothesis test had been conducted in around 60% of the cases. Moreover, although it is a computationally demanding task, another good practice would be to evaluate the generalization ability of an algorithm on an independent test, either by performing k-fold cross validation in supervised learning tasks or testing on different versions of a reinforcement learning task and/or with different initial conditions. Reporting on the values of a fitness function might be valuable in specific cases but it is not necessarily useful in all scenarios, since different definitions of fitness functions can result in different outcomes. Regarding the efficiency of a method, the number of generations until convergence or until the best solution is found (maximal or minimal fitness value) should be reported as well. Reporting on the computational time can be informative but depends on the computational power of a computing machine as well as on the employed processing schemes and it does not allow for direct comparisons among different methods. x-NEAT methods that evolve the topology of an ANN, like the ones extending NEAT and the ones that automatically configure the substrate of the HyperNEAT-based methods should be additionally evaluated on the size of evolved networks, that is, the number of connections and hidden nodes. Agreeing on a set of performance metrics does not exclude the use of other extra metrics if necessary for characterizing the performance of a specific property of an algorithm. Although we stress the importance of comparisons and benchmarking, this does not mean that this is always the most important aspect of an algorithm (Stanley, 2018). Papers that propose new research directions or provide insights are of great value as well. As it is stated in Stanley (2018), “While a good algorithm is sometimes one that performs well, sometimes a good algorithm is instead one that leads to other algorithms and new frontiers.”

Datasets.  Although some datasets are commonly used among x-NEAT methods of the same or of different subclusters such as the XOR, the pole balancing, the retina classification problems, etc., in general x-NEAT methods belonging to the same cluster are compared on different problems. Combined with the different employed performance metrics as illustrated above, it is evident that direct comparisons among different methods become very difficult. More standardization and the use of benchmark problems is required. This article provides an overview of the datasets used in the methods included in the review but an analysis of the datasets and the proposal of a minimal set that is suitable for benchmarking should be addressed in future work.

Parameter Configuration.  All x-NEAT methods depend on a large number of parameters. The majority of the methods presented in this review article report on the employed parameters, while the rest of the papers either include a small portion (e.g., the size of the population or the allowed number of generations) or do not report them at all. However, these parameters are usually configured by trial and error or by reusing previously defined default parameter settings and not by systematically searching the parameter space using intelligent optimization methods. Another issue concerns how parameters should be configured when a method is compared to the state of the art. Optimizing the parameters for the proposed method and then comparing to the old method may cause a bias toward favoring the new method. Optimizing the parameters of the old method and using these on the new method may cause less bias. On the other hand, optimizing the parameters of the two methods separately and comparing the methods on their best performance seems to be the right option. However, it is not very often observed in literature.

The main advantage of a systematic review is that it is performed based on a protocol that is reproducible. Moreover, deciding which papers should be included is based on clearly defined criteria and evaluation scores and not on the authors' opinions. However, one limitation is that the decision of whether a paper should be excluded is based on its abstract. Therefore, papers whose abstracts or keywords did not include a reference to NEAT or even the fact that they are extending NEAT, are not included in this review. Moreover, some other methods can exist in databases other than the Web of Science or Scopus, although most of the high-quality papers are included in these two. We should therefore note that Table 6 does not include an exhaustive list of all the methods that belong to each cluster. Other methods may exist that can either be based on NEAT or employ algorithms independent of NEAT, but they are not included because they do not fulfill the criteria raised by the protocol. Additional methods have been published after the day we last searched the databases on 18 April 2018 and these are not included in this review article. A search performed in Web of Science and Scopus on the 18 July 2019 for papers published after January 2018, resulted into 35 publications. Applying only the inclusion/exclusion criteria and not the evaluation criteria resulted in 17 relevant papers. These concern x-NEAT methods belonging to the three clusters defined in the paper, for example, multiobjective optimization (Ihara and Kato, 2017; Künzel and Meyer-Nieberg, 2018; Chidambaran et al., 2018), evolving ANNs for sequential tasks with memory requirements (Merrild et al., 2018; ElSaid et al., 2019), hybrid NE methods (Nadkarni and Neves, 2018; Peng et al., 2018; McDonnell et al., 2018), optimizing ANNs with different types of nodes (Papavasileiou and Jansen, 2017; Sboev et al., 2018), optimizing deep learning structures (Desell, 2018), etc.

In the graph of Figure 2 we present the number of published papers in WoS and Scopus following NEAT in 2002. We chose to show not only the number of methods proposed to extend NEAT in the ways we present in this article and in the Appendix, but also the number of papers that are applications of an x-NEAT method to a challenging problem. Eighteen years after NEAT we can still see its relevance as the number of published papers extending or applying x-NEAT methods seems to increase.

We believe that especially nowadays in the era of deep learning, NEAT-based algorithms can contribute significantly to the configuration of the DNN. This potential is already evident in literature as hybrid methods of deep learning and NE have emerged. The papers (Schrum, 2018; Miikkulainen et al., 2019; Costa et al., 2019) are examples of x-NEAT methods for evolving DNNs. Especially, DeepNEAT and CoDeepNEAT (Miikkulainen et al., 2019) are two very promising x-NEAT methods (305 citations in Google Scholar) for optimizing the structure of DNNs. DeepNEAT follows the principles of standard NEAT, that is, an initial population of chromosomes, represented by graphs is complexified during evolution by gradually adding structure (edges and nodes) through mutation operators. Crossover is facilitated by historical markings and speciation is applied to protect innovation. The differences between NEAT and deepNEAT are: each gene in the chromosome represents a layer of a DNN, with a table describing its hyperparameters, instead of representing a node of an ANN. Moreover, the edges in the chromosome do not include weights but only indications of the layers' connectivity. During fitness evaluation the chromosome is decoded into a DNN that is trained for a fixed number of epochs. Coevolution DeepNEAT (CoDeepNEAT) is an algorithm optimizing the structure of DNNs inspired by principles of SANE (Moriarty, 1997), ESP (Gomez and Miikkulainen, 1999), and CoSyNE (Gomez et al., 2008). CoDeepNEAT generates repetitive modular structure by evolving two populations of modules and blueprints with coevolution. Each blueprint is a pointer to a module representing a small DNN. These DNNs are combined to create a larger network. We have the opinion that coevolution of two populations or within the individuals of one population is a powerful technique that NEAT-based successors should consider in the future, especially when aiming into reaching open-ended evolution (Stanley et al., 2017).

Quality Diversity (QD) algorithms also have potential applications to open-ended problems (Pugh et al., 2016; Wang et al., 2019). QD algorithms search a behavioral space to find a large set of solutions that are simultaneously diverse and high-performing (Cully and Demiris, 2017). We believe that novelty search with local competition (NSLC) (Lehman and Stanley, 2011b) and multidimensional archive of phenotype elites (MAP-Elites) (Mouret and Clune, 2015) are two QD algorithms that constitute another interesting research area where NEAT-based successors could focus on in the future.

NEAT is a complex method coming from intertwining three principles. Ablation studies showed that each component of NEAT is crucial to its performance (Stanley and Miikkulainen, 2002b). However, since then, research has provided many powerful works such as quality diversity methods (Lehman and Stanley, 2011b; Mouret and Clune, 2015), age fitness Pareto optimization (Schmidt and Lipson, 2011), and others that might raise the question of revisiting these ablation studies and investigating whether each component is really necessary when NEAT is applied in these types of cases. For example, in Mouret and Doncieux (2012), a NE algorithm that employs only the encoding part of NEAT, NEAT's structural mutation operations (no crossover) and NSGA-II (Deb et al., 2002) with an objective rewarding exploration, performs better.

Eighteen years after NEAT's invention, a plethora of successors have appeared. Our review protocol identified 60 methods that met the inclusion criteria and passed the quality assessment threshold. These methods allow us to identify relevant features, perform real time and/or online evolution, optimize multiple objectives, accelerate the computational performance, increase the interpretability of the evolved networks, and allow applications on search spaces with issues including high dimensionality and deceptive landscapes. NEAT is extended by modifications that include: introducing new mutation operators, evolving different types of neural nodes, adding new fields in NEAT's direct encoding, introducing new types of encoding (e.g., developmental), applying different selection strategies, changing the way by which the population is evolved (e.g., evolving subnets, coevolving two populations), etc. As supplementary material, readers can find a graphical tool on our website³ that uses the 18 subclusters defined in this article to guide them in choosing a method that satisfies their needs. This article and the graphical tool can be therefore used as an index to the Appendix of this article where readers can find the description of the method and its reference to the original paper.

To facilitate benchmarking and comparisons, we believe that methods belonging to the same subcluster should be evaluated on the same tasks and should report on the same performance metrics. As a future work, it would be relevant to categorize the datasets/tasks of each subcluster to propose a set of benchmark tasks. Combined with the outcome of this article, researchers will then have all the relevant information to benchmark and compare a new method using the same state-of-the-art benchmark test sets and the same performance metrics. Finally, we hope that with this article we can stimulate similar review papers in other areas of EC, for example, in multiobjective optimization for NSGA-II (Deb et al., 2002) or in Evolutionary Strategy for CMA-ES (Hansen and Ostermeier, 2001).

Evgenia Papavasileiou was funded by a PhD grant from the Research Foundation Flanders (FWO) (scholarship numbers 1S34016N, 1S34018N).

Principles.  Coevolutionary NEAT (Stanley and Miikkulainen, 2004) is applicable to open-ended problems where the final solution is not known and more sophisticated strategies can be continuously generated using coevolution. A coevolutionary algorithm is an EA in which an individual's fitness cannot be evaluated based on an objective measure, but instead based on interactions with the other individuals of the population. It modifies the NEAT algorithm to evolve two separate populations that are evaluated against each other. Each population is evaluated against a sample of the opponent population consisting of the champions of the best four species and eight champion solutions of the past generations that are archived into a so-called “Hall of Fame.” In this way, it is guaranteed that the opponent sample consists of good and diverse solutions and that the current solutions are competitive against older solutions. NEAT's complexification ability is leveraged to allow continual coevolution, that is, finding new solutions while maintaining existing capabilities, by elaborating on existing strategies. To achieve a sequence of increasingly more sophisticated strategies, a dominance tournament method is defined (Stanley and Miikkulainen, 2002a) so that champion strategies in every generation are evaluated against previous dominant strategies.

Encoding.  NEAT's encoding is used.

Evaluation.  The evolved strategies are evaluated on their complexity, that is, the number of nodes and connections that exist in the dominant network averaged over the different runs. A dominance tournament method (Stanley and Miikkulainen, 2002a) is defined to evaluate the progress of evolving more sophisticated strategies. The winner strategy of a generation is considered dominant if it defeats all the previous dominant strategies. Coevolutionary NEAT is evaluated against fixed topology coevolution and simplifying coevolutionary NEAT, by direct applications in a simulated robot duel. A simplifying coevolutionary algorithm works in a way opposite a complexifying one, that is, the evolution starts with a complex network and structure is gradually removed. The algorithms are compared using the highest achieved dominance level and the generation when dominance occurred. Moreover, the champion strategy of the simplifying and fixed-topology coevolutionary algorithms is evaluated relative to the performance of coevolutionary NEAT by comparisons against its entire dominance ranking. This metric is calculated by taking the dominance level of the highest strategy of coevolutionary NEAT that is defeated by a strategy from the champion of the other coevolutionary algorithm and dividing by the total number of dominance levels. Finally, the equivalent generation in which the latter happens is also considered a metric. It is concluded that coevolutionary NEAT with complexification finds more sophisticated strategies than fixed topology coevolution and simplifying coevolutionary NEAT.

Principles.  Modular NEAT (Reisinger et al., 2004) is a TWEANN coevolutionary algorithm that enables the simultaneous evolution and modularization of network topologies. It breaks the problem into subproblems by evolving neural substructures (from one neuron to whole networks) that can be reused and combined together into more complex structures. The algorithm is suitable for high-dimensional search spaces.

Modular NEAT consists of two types of populations: a population of modules (NEAT networks) and a population of blueprints. Each module is a set of links connecting abstract input, output, and hidden nodes. The set of blueprints defines how the modules can be combined. Each blueprint consists of a fixed-size list of pairs of modules and mappings from the abstract nodes to the nodes of the solution, a process called binding. Both populations are evolved together symbiotically, such as in the SANE method (Moriarty, 1997). Symbiotic evolution is defined in Moriarty (1997) as “a type of coevolution where individuals explicitly cooperate with each other and rely on the presence of other individuals for survival.” The populations of modules and blueprints can evolve in different time scales and in two sequential phases, each of which include procedures of selection, crossover, mutation, and fitness evaluation.

Encoding.  The module population is encoded as in NEAT, while historical markings are also assigned on the blueprint population.

Evaluation.  Modular NEAT is tested in an artificial board game with two functionally different regions and varying input/output dimensionality. Its ability to produce modular networks is measured by the percentage of crosslinks that connect an input from the one half of the board game to the output of its other half. Modular NEAT is compared to NEAT and it can obtain better networks with higher average accuracy achieved in less generations and with less crosslinks.

Principles.  Feature Selective NEAT (FS-NEAT) (Whiteson et al., 2005) extends NEAT by performing feature selection in addition to the learning of the network's topology and weights. In FS-NEAT, evolution starts even more minimally than in NEAT, with networks with one random input connected to one random output. As evolution progresses, new inputs, connections and hidden nodes can be added as in NEAT. FS-NEAT performs implicit feature selection by maintaining those connections that initiate from the inputs which enhance the network's performance, that is, the relevant inputs.

Encoding.  NEAT's encoding is used.

Evaluation.  FS-NEAT is evaluated on a RL task of racing robot cars and it is compared to NEAT in terms of fitness performance, number of generations, and size of the evolved networks. It is found that FS-NEAT can learn better networks than NEAT in less generations especially when redundant or irrelevant inputs are present in the input dataset.

Principles.  Real-Time NEAT (rtNEAT) (Stanley et al., 2005) allows evolving video games' agents in real time. Each agent is represented by a NEAT-evolved ANN. In this way the agents can develop increasingly complex behavior which can be changed and be improved throughout the game, guided by the player itself. The main difference with NEAT lies in the reproduction cycle; in NEAT the whole population is replaced by a new population at each generation. But, if this would happen in a real-time environment, such as in a video game, all the agents would instantly change their behavior. In order to avoid this, rtNEAT replaces only one of the worst individuals of the population with the offspring of two of the best parents. This replacement occurs continuously and becomes unnoticeable by the player. To achieve speciation in real time, NEAT's reproduction cycle is changed and a new agent is introduced into the population every $n$ ticks of the clock, to remove the agent with the worst adjusted fitness, given he has been alive for a minimum time.

Encoding.  NEAT's encoding is used.

Evaluation.  rtNEAT's abilities are illustrated on a military combat game called NERO in which the user can train robots to develop skills suitable for battle. Therefore, its performance is subject to the user's perception of when the trainable agents have reached the desired skills.

Principles.  rtNEATv2 (D'Silva et al., 2005) extends rtNEAT (Stanley et al., 2005) to train agents that should show different behaviors and perform multiple tasks. Milestoning refers to keeping a pool of old individuals, drawn from the population of agents at a specific time, that know how to perform a given task. With a specific probability and tournament selection, an individual, which at the point of the pool's creation had high fitness, is selected from the pool. Another individual is selected from the current population and the two are mated in order to produce a hybrid offspring. The generated offspring has the ability to perform both tasks, as it contains genes that come from both its ancestor and its modern parent.

Encoding.  NEAT's encoding is used.

Evaluation.  rtNEATv2 with different parameter settings regarding the size of the pool and the probability of creating a hybrid offspring are evaluated on the NERO task and compared against rtNEAT (Stanley et al., 2005) (Section A.4). The agents are trained on two tasks in two subsequent phases and they are tested on a third phase to evaluate whether they remember the first task or not. To evaluate their performance, the distance remaining from the target, as well as on the portion of agents from the population that solved each of the two tasks successfully are used.

Principles.  CPPN-NEAT (Stanley, 2006) is a method using NEAT to evolve Compositional Pattern Producing Networks (CPPNs). CPPNs are proposed as a new abstraction of encoding natural development by simulating it through compositions of functions based on observed gradient patterns in natural embryos. CPPNs are connected graphs of functions, therefore they resemble ANNs with nodes of different activation functions (Gaussian, sigmoid, and periodic). Because CPPNs look like ANNs, NEAT can be used to complexify them by evolution. In CPPN-NEAT, evolution is interactive, that is, during the selection step the parent population is selected by the user. In contrast to NEAT, the population of CPPNs is initialized with one hidden layer of one or two hidden nodes and speciation's similarity metric is changed to take into account the different activation functions.

Encoding.  CPPN-NEAT modifies NEAT's encoding by adding a field in the node genes to specify the nodes' activation functions.

Evaluation.  CPPN-NEAT's performance is not evaluated based on a metric, but based on visual interpretation of the evolved patterns.

Principles.  NEAT+Q (Whiteson and Stone, 2006a) is an evolutionary function approximation algorithm, that is, a hybrid method using NEAT to configure the ideal ANN that can learn to represent the value estimates provided by Q-learning, a Time Difference (TD) method. During NEAT's fitness evaluation, which is repeated for $e$ episodes, the ANNs' weights are updated by backpropagation, with each episode starting with the weights of the last episode. Two approaches that enable the synergy between evolution and learning are applied: Lamarckian and Darwinian. With the former, any changes made during a generation are written back to the genome and inherited by the offsprings, while with the latter these changes are discarded.

NEAT+Q is further extended to online NEAT+Q for online scenarios where an agent aims at learning a good policy quickly while maximizing its reward (exploration and exploitation). For this purpose, two action selection mechanisms that can balance these two objectives are used: $ε$-greedy and softmax. These selection mechanisms are applied at the level of policies evaluation, aiming at maximizing the reward of the best policy and increasing the average reward, respectively.

Encoding.  NEAT's encoding is employed.

Evaluation.  The performance of NEAT+Q and online NEAT+Q is evaluated using the uniform moving average score per episode averaged over the different algorithmic runs. It was found that NEAT+Q outperforms TD methods, while online NEAT+Q improves EC's online performance.

Principles.  Layered Pareto Coevolution Archive NEAT (LAPCA-NEAT) (Monroy et al., 2006) is a Pareto coevolutionary algorithm that uses NEAT to evolve a population of individuals that competes against itself. Pareto coevolution is treated as a multiobjective optimization problem whose candidate solutions are known as learners and the individuals on which they are tested as testers or evaluators. In Pareto coevolution, the learners are compared in terms of Pareto dominance to a set of testers that are viewed as the objectives being optimized. LAPCA-NEAT's coevolutionary memory is a LAPCA (de Jong, 2004) that consists of nondominated learners organized in layers from past generations drawn from the species' champions set and testers that are able to discriminate between layers. This enables users to keep the number of evaluations that are required to establish a Pareto dominance low.

Encoding.  NEAT's encoding is used.

Evaluation.  LAPCA-NEAT is tested against other types of memories, for example, a Hall of Fame (HOF) as coevolutionary memory on the pong game. Two sets of comparisons are performed: the best of run and the best of generation to evaluate which coevolutionary memory evolves better players and which one results in better coevolutionary progress, respectively. The performance of a player is determined by the number of players it dominates, whereas the coevolutionary progress is evaluated based on the average number of wins, ties and losses. Finally, the two memories are compared based on the size of the archive they maintain. It is found that LAPCA uses less memory but it does not outperform a Hall of Fame memory in terms of the players' performance.

Principles.  Learning-NEAT (L-NEAT) (Chen and Alahakoon, 2006) is a hybrid learning algorithm applied to classification problems. It combines NEAT with backpropagation to benefit from global and local search, respectively. L-NEAT (Chen and Alahakoon, 2006) employs a divide-and-conquer approach to divide each network of the population into fully connected subnetworks with only one output node corresponding to different subtasks. The final solution is generated by assembling the networks that perform best in each subtask. With this approach, the computational expensive problem of each individual's fitness evaluation is addressed. Moreover, NEAT's purpose is changed from searching the optimal solution to searching for networks that will correspond well to backpropagation, called feasible points. In this case, backpropagation takes place every few generations only on the NEAT-evolved networks with high fitness values.

Encoding.  Since NEAT is employed during the global search, the same encoding as in NEAT is used.

Evaluation.  The classification accuracy averaged over twenty runs is engaged for the evaluation of L-NEAT's (Chen and Alahakoon, 2006) performance. L-NEAT outperforms both NEAT and NEAT enhanced with a divide-and-conquer method.

Principles.  Nnrg.hazel (Reisinger et al., 2007) is a coevolutionary algorithm based on NEAT for evolving strategies for General Game Playing (GGP). In GGP, agents can play a variety of unknown games described in first order logic. With coevolution the space of policies and multiple opponent strategies can be searched with little a priori knowledge. In nnrg.hazel, the ANNs that are evolved by NEAT are used as heuristic evaluators in a minimax partial game tree search. As in Stanley and Miikkulainen (2004), two populations of strategies are evolved and the fitness of an individual is evaluated against a group of opponents. To ensure that more sophisticated strategies will be discovered and avoid recycling behavior, the Covering Competitive Algorithm (CCA) (Rosin and Belew, 1997) is employed. This technique, similar to the dominance tournament algorithm in Stanley and Miikkulainen (2004), guarantees that monotonic progress toward some Pareto optimal solution will be made. During evolution, the individuals of the two populations are competing against a random set of opponents of the current generation but also against previous strategies. This is achieved through maintaining a set of previous dominating strategies called teachers. In each generation, each individual of a dominated population, that is, a population whose strategies are beaten by the opponent's teachers, is evaluated against the teachers of the other population in order to determine which population will dominate which.

Encoding.  NEAT's encoding is used.

Evaluation.  Nnrg.hazel is tested on a set of games from the GGP using only 1-ply search and it is compared against random search (also 1-ply). The performance metrics used include the average size of the teachers' set (representing the best fitness achieved) and the average match score plotted against the teacher's rank. To evaluate scalability of the 1-ply evolved heuristics to a deeper n-ply search, the evolved heuristics are used as n-ply evaluators (⁠$n∈[1,4]$⁠) and their performance is measured by the average game score. Finally, the average amount of generations that each teacher with a specific role is not dominated is computed, to give an estimation of disengagement, that is, “when one population outperforms the other to the point that there is no longer a clear gradient that selection can flow” (Reisinger et al., 2007). It is found that nnrg.hazel outperforms random search but cannot scale in deeper search depths and it shows some degree of disengagement.

Principles.  Kernel-based Online NEAT (KO-NEAT) (Zhao et al., 2007) is a hybrid NE algorithm for improving NEAT's online performance. It borrows an action selection mechanism from TD algorithms called Interval Estimation (Taylor et al., 2006) to take advantage of the fitness information gained in earlier generations so that more promising individuals will be assigned more episode evaluations. Moreover, KO-NEAT employs $μ+λ$ evolutionary strategy to create the next generation's population from the combined $μ$ parents and $λ$ children based on their fitness. Finally, the offsprings' initial fitness is calculated using kernel function approximation.

Encoding.  NEAT's encoding is used.

Evaluation.  KO-NEAT (Zhao et al., 2007) is compared to NEAT on the keepaway soccer game (Stone et al., 2005) using the maximum fitness acquired by the best individual as performance metric. KO-NEAT was able to plateau sooner, accumulate more rewards during the same episodic evaluations and learn a better policy than NEAT. In conclusion, KO-NEAT improves NEAT's online performance and accelerates learning efficiency and speed.

Principles.  NEAT-CTRNN (Miguel et al., 2008) uses NEAT to evolve Continuous Time Recurrent Neural Networks (CTRNN). CTRNNs are a type of ANN with recurrent connections and nodes equipped with an internal time constant $τ$ and whose response in the incoming neuron spike is based on differential equations. These types of nodes belong to a class of dynamic neuron models which are considered to be more realistic models of biological neurons.

Encoding.  NEAT's encoding is used.

Evaluation.  The algorithm is evaluated based on its generalization ability and the number of fitness evaluations. At the end of each generation, the best network is tested on a separate test with different initial conditions. The generalization score is defined as the fraction of times the network was able to solve the task. It is found that NEAT-CTRNN evolves smaller networks and requires fewer evaluations compared with other NE methods that evolve traditional ANNs.

Principles.  HyperNEAT (Stanley et al., 2009) extends CPPN-NEAT (Section A.6) for designing large-scale ANNs of fixed topologies. It evolves the connectivity of ANNs by exploiting information of the geometry of the application domain. This can be achieved by defining the ANNs as substrates with input and output nodes placed in a way that is representative of their original positions. In this way, substrates with various topologies suitable for solving different problems can be defined, such as a (2D/3D) grid, a sandwich or a circular substrate.

Encoding.  HyperNEAT uses developmental (indirect) encoding to map the regularities of the task's geometry onto the ANN's topology. Developmental encoding allows the reuse of the same genes for developing multiple structures. It is inspired by the biological encoding between the genome and the phenotype. NEAT is used to evolve the topology, weights, and activation functions of CPPNs which output a spatial pattern in the hyperspace. This pattern corresponds to a connectivity pattern in the substrate, that is, the CPPNs output the weight of the connection of the queried points of the substrate.

Evaluation.  HyperNEAT's performance is compared to a modification of NEAT called Perceptron NEAT (PNEAT) (i.e., NEAT for evolving perceptrons) on a task of object recognition in a 2D space. As a metric, the average distance between the champion's output and the real position of the target at each generation, across all trials and runs is used. It is found that HyperNEAT is able to find better solutions and has better generalization ability than PNEAT. Moreover, HyperNEAT, in contrast to PNEAT, is able to scale to higher resolutions without the need of further evolution.

Principles.  Transfer Learning CPPN-NEAT (TL-CPPN-NEAT) (Bahçeci and Miikkulainen, 2008) is a hybrid method that proposes evolving patterns as game heuristics for GGP tasks and transferring the learned patterns on other tasks. With transfer learning, the heuristics learned on a source task can be transferred to a similar, slightly more difficult target task. In TL-CPPN-NEAT, patterns are evolved by CPPN-NEAT (Section A.6) and transfer learning is achieved by using the final population evolved on a source task to initialize the population to be evolved on a target task. In contrast to CPPN-NEAT, the CPPN nodes include only sigmoid and Gaussian activation functions and a new mutation operator is introduced to change the activation function from Gaussian to sigmoid (or from sigmoid to Gaussian) without changing the innovation number of the node. This allows improving individuals that are near the optimal solution, but one of their nodes has the wrong activation function.

Encoding.  The method is an extension of CPPN-NEAT, therefore it uses NEAT to evolve CPPNs and NEAT's direct encoding to encode the genome.

Evaluation.  The method is tested on a single-player board game with one version serving as the source task and a more complex version as the target task. TL-CPPN-NEAT is evaluated against CPPN-NEAT (Section A.6) without transfer learning. The metrics used include the player's performance on the game, given by the ratio of successful checks and the total learning time, given by the number of generations. It is found that transfer learning is beneficial, improving the performance and the learning time.

Principles.  Multiagent HyperNEAT (D'Ambrosio and Stanley, 2008) extends HyperNEAT for applications in multiagent domains, where the agents specialize in specific tasks and therefore are characterized by separate policies. However, policies can also share common characteristics and be complementary. Multiagent HyperNEAT enables the automatic discovery of policies, as a pattern with common characteristics and variations. With multiagent HyperNEAT, there is no need to separately define each agent's policy nor rediscover the common features. This pattern of policies is produced as a function of the agents' position in the field through a single genome described in the following paragraph. Multiagent HyperNEAT is further modified by seeding the evolution with prior knowledge in the form of adding a good genome in the initial population.

Encoding.  Multiagent HyperNEAT uses a modified version of CPPNs as indirect encoding, where two substrates need to be defined; the first one is an ANN substrate describing the geometry of the sensors of each agent while the second one describes the position of the agents in the field. In this way, the second substrate consists of a set of substrates whose connection weights are calculated as a function of their location within each network and within the larger substrate. Using a single genome, a heterogeneous team of agents, each represented by the same ANN substrate, can be evolved. As in HyperNEAT, a population of CPPNs is evolved that takes as inputs the positions $xi,yi$ of each $i$ agent. However, the CPPNs initial topologies are altered to include a hidden node $r(x)$ after the input node of the $xi$ coordinate. With this modification, the $x$ coordinate frame is repeated for each agent in order to mark positions of each agent. In the end, a single CPPN can encode the outputs of the agents as patterns that share characteristics but also show variation.

Evaluation.  Multiagent HyperNEAT with and without seeding that evolves heterogeneous groups of agents is evaluated against multiagent HyperNEAT that evolves homogeneous groups of agents (i.e. against HyperNEAT), with and without seeding. These algorithms are evaluated on a predator–prey task with a varying number of preys and a variety of topological prey formations repeated several times. The time remaining after the agents have captured all the preys is used as a performance metric, finding multiagent HyperNEAT with seeding to perform the best among the tested algorithms.

Principles.  Content-generating NEAT (cgNEAT) (Hastings et al., 2009) is an interactive EC method that uses a modified version of NEAT to evolve CPPNs for automatic content generation in video games, as they are played, based on each player's behavior. cgNEAT is based on NEAT but it differs in some points. To begin with, the population size is not a developer-defined parameter but it depends on the number of players in the game. Moreover, the selection of the “bad” behaving networks is done implicitly by the players' actions and when an offspring is created it is not placed immediately in the population but it stays in a temporary state from which the player can decide whether it will be added in the population or not. Finally, no speciation exists and diversity is maintained based on the player's preferences.

Encoding.  The generated content includes weapons which are evolved through particle systems used in computer graphics for producing non-solid or fuzzy phenomena such as fire, smoke, water, explosions, etc. These particle systems are encoded by CPPNs and evolved with NEAT.

Evaluation.  cgNEAT is evaluated on the Galactic Arms Race (GAR) multiplayer video game using statistics generated by the players and the evolved content. cgNEAT is able to evolve a variety of unique and tactically diverse weapons, but it is not compared with the state of the art.

Principles.  Reinforcement Learning using State Aggregation via NeuroEvolution (RL-SANE) (Wright and Gemelli, 2009) is a hybrid method combining NEAT and Sarsa(⁠$λ$⁠) (Sutton and Barto, 1998). It combines ANNs' advantage of being good function approximators, able to abstract knowledge and RL's ability to explore and exploit this knowledge. In particular, RL-SANE uses the custom ANNs, evolved by a NEAT version that allows random pruning [Blended Searching (BS-NEAT) (James and Tucker, 2004)] as a means to reduce the state space, by aggregating state information into a more compact representation, so that Sarsa(⁠$λ$⁠) can be applied to problems with large state space. The generated ANNs map inputs from a raw multidimensional space to a one-dimensional continuum consisting of different discrete states. The ANNs output the state identifier which is provided to Sarsa(⁠$λ$⁠) as an index to its Q-table which keeps track of the values of different actions taken in specific states.

Encoding.  RL-SANE uses NEAT's encoding.

Evaluation.  RL-SANE is compared to BS-NEAT (NEAT with structure pruning) (James and Tucker, 2004) based on the average number of time steps the most fit members of the population were able to solve two state-of-the-art RL problems. It is found that RL-SANE performs better than NEAT in both domains and evolves less complex networks. However, this is anticipated since in RL-SANE, NEAT evolves ANNs which perform only state aggregation whereas NEAT's ANNs perform both state abstraction and policy iteration which is a much more complicated task. These findings suggest that RL-SANE can scale to more complicated domains.

Principles.  Feature De-selective NEAT (FD-NEAT) (Tan et al., 2009) extends NEAT in terms of performing feature selection simultaneously with optimizing the topology and the weights of ANNs. FD-NEAT differs from FS-NEAT (Section A.3) in the way feature selection is performed. FD-NEAT starts with the same minimal topology as NEAT, i.e. with all the input nodes connected to the output nodes. In order to perform feature selection, a new mutation operator is introduced that is able to drop connections initiating from the input layer. Implicit feature selection is therefore performed as the GA can drop connections from irrelevant or redundant inputs that decrease the performance and keep the relevant ones that increase it.

Encoding.  NEAT's encoding is used.

Evaluation.  FD-NEAT is compared to NEAT and FS-NEAT based on the maximum fitness achieved by the best network, the number of generations required to obtain this score and the number of evolved connections averaged over the different algorithmic runs. To assess the feature selection ability two metrics are used: the frequency with which the population selects the relevant inputs and the sum of the absolute values of the connections' weights that originate from each input in the population's best network. It is found that FD-NEAT achieves the maximum fitness score in less time than FS-NEAT and it offers the best compromise between the size of the evolved networks and their performance.

Principles.  Radial Basis Function NEAT (RBF-NEAT) and Cascade-NEAT (Kohl and Miikkulainen, 2009) are two algorithms extending NEAT, proposed to deal with fractured search spaces, where an agent's actions change discontinously among states, by capturing the problem's local features.

RBF-NEAT (Kohl and Miikkulainen, 2009) is proposed as a method that allows local adjustments to policies, therefore it is suitable for discovering solutions that are discontinuous and can change dramatically. RBF-NEAT extends NEAT by allowing two types of activation functions in the evolved ANNs: sigmoid and radial basis functions. To do that a new mutation operator called “add RBF node” is introduced which enables the introduction of nodes with Gaussian activation functions. The parameters of the Gaussian function are controlled together with all the other parameters by the GA of NEAT. The new algorithm enables exploration of the space through the normal NEAT mutation operators and in addition it introduces a bias toward local-processing structures through the RBF node mutations.

Cascade-NEAT (Kohl and Miikkulainen, 2009) constrains the evolved topologies into regular, cascade structures that can make local refinements, by adding a new mutation operator called “add cascade node.” When such a node is introduced, it receives connections from all the inputs and the existing hidden nodes and is connected to all the outputs while it freezes all the preexisting connections to allow mutating only those connections that concern the recently added hidden node.

Encoding.  NEAT's encoding is used.

Evaluation.  The two proposed algorithms are compared against each other, against NEAT and a NEAT version that does not allow structural mutations on problems with variation. Variation is proposed (Kohl, 2009) as a means to estimate how fractured a problem is, by treating a solution as a function and calculating its variation. On a problem of generating solutions with maximal variations, Cascade-NEAT is able to produce significantly higher variation, while RBF-NEAT is more effective when the number of inputs is low. The algorithms are also compared on other benchmark supervised and RL problems in terms of fitness. It is observed that Cascade-NEAT and RBF-NEAT have significantly higher performance than NEAT; however, their scores decrease as the variation of the problem increases.

Principles.  Multiobjective NEAT (MO-NEAT) (Haggett and Chu, 2009) is a NEAT version for dealing with problems with many conflicting objectives. MO-NEAT differs from NEAT as the fitness function assigns one value for each objective and fitness sharing is calculated for each objective. Moreover, it uses the NSGA-II algorithm (Deb et al., 2002) as the parent selection mechanism for reproducing, while the offsprings are assigned to species according to the value of the shared fitness of two objectives. MO-NEAT evolves ANNs to act as novelty detectors. Any significant deviation in the system's behavior is defined as novelty. The evolved ANNs adopt principles from Dynamic Predictive Coding (DPC) (Hosoya et al., 2005), to allow online learning by continuously updating the networks' weights. The hidden and output nodes can have two types of connections: connections with fixed weights determined by evolution and modifiable connections whose weights are modified by an anti-Hebbian rule determined by DPC parameters that are also subject to evolution and determined by a new mutation operator called “mutate Parameters.”

Encoding.  NEAT's encoding is altered by adding a new field “connectionType” to the connections' genes that determines whether a connection is modifiable or not.

Evaluation.  MO-NEAT as a novelty detector is tested on two tasks which are successfully solved, while it outperforms the results in Le Borgne et al. (2007) and Sohn et al. (2001) based on metrics specific to the used datasets and on the percentage of true positives and false positives.

Principles.  Adaptive HyperNEAT (Risi and Stanley, 2010) extends HyperNEAT (Section A.13) to concurrently evolve patterns of connectivity and learning rules as a function of geometry, allowing the evolution of plastic ANNs. In this way, the synaptic weights of the ANNs' connections do not remain static during the networks' lifetime but they have the ability to change in response to the changing activation levels in the neurons they connect. The advantage of this method is that the rules are derived from the network as a function of geometry and they do not have to be explicitly defined.

Encoding.  Adaptive HyperNEAT encodes the connectivity patterns and the learning rules with indirect encoding using three modified versions of CPPNs. In the first case, known as iterated model, three additional inputs are introduced in the input layer of the CPPN that represent the current connection weight $wij$ between two nodes $i$ and $j$⁠, the presynaptic activity of node $i$⁠, $oi$⁠, and the postsynaptic activity of node $j$⁠, $oj$⁠. In the second case, known as Hebbian ABC model, the CPPN's output layer is augmented with four additional outputs in order to output not only the connection weight but also the learning rate $η$ and three parameters $A,B,C$ for defining the learning rule as $Δwij=η[Aoioj+Boi+Coj]˙$⁠. In the final case, only one additional output representing the learning rate $η$ is placed on the CPPN's output layer and the rule of plasticity is given by $ΔWij=ηoioj˙$⁠.

Evaluation.  The performance of Adaptive HyperNEAT is evaluated on two navigation scenarios that require the agent to adapt in order to find the position of a reward. The employed evaluation metrics include the number of generations required to find a solution as well as the maximum fitness eventually reached. No comparisons exist between Adaptive HyperNEAT and other NE methods.

Principles.  Online NEAT (Whiteson and Stone, 2006b) and online rtNEAT (Reeder et al., 2008) are two methods for online evolution extending NEAT and rtNEAT, respectively. In contrast to offline evolution where agents are trained for once and then applied on a computer game, online evolution enables the evolution of agents with the ability to adapt. Moreover, online evolution focuses on maximizing the performance of an agent during the whole learning process in contrast to real-time evolution which focuses on quickly finding a group of agents that perform well. In onlineNEAT (Whiteson and Stone, 2006b) and online rtNEAT (Reeder et al., 2008), the fitness is evaluated on a number of episodes and it is assigned to each individual according to their performance so that better individuals are evaluated for more episodes. Toward this goal, different selection strategies from the RL domain are adopted including $ε$-greedy search, softmax and interval-based techniques. The authors of Cardamone et al. (2010) modify these algorithms by including an additional evaluation strategy called $ε$-greedy improved (Cardamone et al., 2009), that changes the fitness evaluation by averaging the fitness values over the different episodes and assigning small time slots to evaluations corresponding to only a portion of the problem instance. Moreover, in Cardamone et al. (2010), the influence of transfer learning by seeding the evolution with the best individuals evolved at different tracks of the game, is investigated.

Encoding.  NEAT's encoding is used.

Evaluation.  Online NEAT and online rtNEAT with different selection strategies are compared against each other, against offline NEAT and rtNEAT (Section A.4) on different tracks of a car simulator. Their evaluation is based on the average lap time of the best individuals at the end of evolution or the average lap time during the first and last laps. The champions are then tested on different tracks of two laps in total and their performance on the second lap is measured. While online rtNEAT seems to perform better than online NEAT in the beginning of evolution, at the end they do not differ statistically and onlineNEAT can generalize better. Moreover, seeding the population with good individuals accelerates the learning process and improves the overall performance.

Principles.  Multiagent HyperNEATv2 (D'Ambrosio et al., 2011) extends Multiagent HyperNEAT (D'Ambrosio and Stanley, 2008) so that agents in a multiagent environment can learn multiple policies as a function of their position (Policy Geometry, PG) and their current state (Situational Policy Geometry, SPG).

Encoding.  Multiagent HyperNEATv2 uses CPPNs as indirect encoding with two different substrates; one ANN describing the individual agents and one describing the team of the agents. As in original multiagent HyperNEAT, the team substrate contains the individual substrates but now these are stacked in $z$ dimension reflecting the position of the agent within the team (PG). Moreover, the team substrate is enhanced with another dimension $S$ to model the different team policies (SPG). Therefore, the substrate consists of multiple stacks of agents, one for each policy. CPPNs' topologies are also changed to include two more input nodes: a node $z$ describing the position of an agent within the team and a node $S$ describing whether the agent should perform another task or not. By incorporating this node in the CPPN, the pattern that is generated is a function of the position of the agent within the team, its current state and its individual parameters.

Evaluation.  Multiagent HyperNEATv2 is compared against its previous version (multiagent HyperNEAT) that evolves policies based on their policy geometry, on a difficult patrol and return task using both simulation and real robots. The performance of the two algorithms is evaluated in terms of whether the robots successfully fulfilled the task and the number of generations. It is found that Multiagent HyperNEATv2 was more consistent in finding solutions and robust enough to be transferred to real world scenarios.

Principles.  Recurrent CPPN-NEAT (Auerbach and Bongard, 2011) is a NE method for designing the morphology of robots and the networks that control their policies. It extends CPPN-NEAT (Section A.6) by allowing recurrent connections in the evolved CPPNs, instead of only feedforward connections. Moreover, the method extends a previous CPPN-NEAT version (Auerbach and Bongard, 2010) that allows evolution to differentially optimize the size and the quantity of evolved robots' components. The CPPNs have outputs that determine the morphology of the components evolved (e.g., the radius of a spherical component and the presence of joints), the placement of sensors as well as the embedded closed loop ANNs (CTRNNs) that control the robot's motion and policies.

Encoding.  NEAT's direct encoding is used to evolve the CPPNs that determine the robots' morphological components. However, the CPPNs are also used as generative encoding that output parameter values of the CTRNNs.

Evaluation.  Recurrent CPPN-NEAT is compared to CPPN-NEAT without recurrency and it evolves more successful robots. The performance metrics used are the fitness function, the size of the evolved CPPNs (number of nodes, connections, forward connections, number of hidden nodes with a specific activation function), as well as several metrics and statistics concerning the evolved morphology (number of spheres, joints, distant sensors, touch sensors, etc).

Principles.  HyperNEAT with Link Expression Output (HyperNEAT-LEO) (Verbancsics and Stanley, 2011) extends HyperNEAT to evolve large-scale ANNs that are characterized by modularity, on top of regularity and hierarchy, as a function of geometry.

Encoding.  HyperNEAT-LEO uses CPPNs as indirect encoding, modified to have two outputs, one that delivers the weight pattern and one that determines whether the connection is expressed (LEO output). With this output, an expression pattern is generated independently of the weight patterns, separating network connectivity from weight patterns. Moreover, modularity can be supported by seeding the CPPNs' initial topologies with a hidden node of a Gaussian function that receives the difference of coordinates so that substrate connections close to each other can be encouraged.