Looking to nature as inspiration, for at least the past 25 years, researchers in the field of neuroevolution (NE) have developed evolutionary algorithms designed specifically to evolve artificial neural networks (ANNs). Yet the ANNs evolved through NE algorithms lack the distinctive characteristics of biological brains, perhaps explaining why NE is not yet a mainstream subject of neural computation. Motivated by this gap, this letter shows that when geometry is introduced to evolved ANNs through the hypercube-based neuroevolution of augmenting topologies algorithm, they begin to acquire characteristics that indeed are reminiscent of biological brains. That is, if the neurons in evolved ANNs are situated at locations in space (i.e., if they are given coordinates), then, as experiments in evolving checkers-playing ANNs in this letter show, topographic maps with symmetries and regularities can evolve spontaneously. The ability to evolve such maps is shown in this letter to provide an important advantage in generalization. In fact, the evolved maps are sufficiently informative that their analysis yields the novel insight that the geometry of the connectivity patterns of more general players is significantly smoother and more contiguous than less general ones. Thus, the results reveal a correlation between generality and smoothness in connectivity patterns. They also hint at the intriguing possibility that as NE matures as a field, its algorithms can evolve ANNs of increasing relevance to those who study neural computation in general.
For at least the past 25 years (Fogel, Fogel, & Porto, 1990; Harp, Samad, & Guha, 1989; Kampfner & Conrad, 1983), a unique and growing research community has focused on the possibility that effective artificial neural networks (ANNs) can be evolved through evolutionary algorithms (Angeline, Saunders, & Pollack, 1993; Floreano, Dürr, & Mattiussi, 2008; Fogel, Fogel, & Porto, 1990; Kitano, 1990; Stanley & Miikkulainen, 2002b; Stanley, D'Ambrosio, & Gauci, 2009; Whitley, 1995; Yao, 1999). This field of research, called neuroevolution (NE), draws inspiration from nature, wherein brains of astronomical complexity are indeed the product of an evolutionary process. Although NE is recognized within the evolutionary computation community, it remains little known within the broader neural network and neural computation communities. Perhaps one reason for this lack of engagement is that although considerable research in neural computation has focused on modeling aspects of real brains (Andrade, Muro, & Morán, 2001; Bednar, Kelkar, & Miikkulainen, 2002; Miikkulainen & Dyer, 1989; Miikkulainen, Bednar, Choe, & Sirosh, 2005; Swindale, 1996), NE algorithms generally do not produce ANNs reminiscent of biological brains. While producing biologically plausible ANNs is a high threshold for success, the suggestive analogy between NE and the evolution of brains in nature invites such ambitious expectations. In fact, it is fair to ask why an artificial evolutionary process should not produce structures reminiscent of those seen in nature. While this question is challenging, it raises the possibility that if we can identify the missing ingredients, it might be possible to incorporate them into future algorithms, thereby dramatically opening up the possibilities encompassed by NE.
In this spirit, this letter takes a step toward artificially evolving more biologically plausible ANNs. Like other work on NE (Gomez, Schmidhuber, & Miikkulainen, 2008; Stanley & Miikkulainen, 2002c; Whiteson, Kohl, Miikkulainen, & Stone, 2003), it includes a performance comparison that demonstrates the advantage of its approach—in this case, in the domain of checkers. However, more importantly the letter concludes with a new kind of analysis that has so far been impossible with evolved ANNs. In the analysis, connectivity patterns of evolved ANNs are shown to exhibit topographic maps. Furthermore, observable geometric properties of these maps are shown to correlate to greater generalization. Thus, for the first time, macrolevel qualitative properties of evolved ANN connectivity patterns are analyzed similarly to networks engineered as cortical simulations (Bednar & Miikkulainen, 2006; Bednar et al., 2002; Choe & Miikkulainen, 2004; Swindale, 1996) or even real cortical layers (Hubel & Wiesel, 1962, 1968; Weliky, Bosking, & Fitzpatrick, 1996). This analysis becomes possible because of the unique properties of the NE algorithm studied in this letter.
Before previewing the details of this algorithm, let us return to the question of the missing ingredient: Why have past NE algorithms not produced ANNs that inspired the attention of neuroscientists or neural computation researchers? For a significant class of NE algorithms called direct encodings (Gomez & Miikkulainen, 1997; Pujol & Poli, 1998; Stanley & Miikkulainen, 2002a; Whitley, 1995), a key reason for the lack of engagement is that they lack geometry. That is, while such evolved ANNs contain an aggregate of nodes and connections, the nodes do not typically exist at defined locations in space, which means that their geometry is in effect arbitrary. There is thus no concept of locality or long versus short connections; there is no concept of symmetry or topographic regularity. Yet geometry is a critical facet of biological brains that is responsible for a preponderance of insight into their workings. Everything from MRI studies of salient brain regions for important cognitive tasks (Rosen & Lenkinski, 2007) to topographic maps (Bednar & Miikkulainen, 2006; Sporns, 2002), such as somatotopic, visual, and auditory, depends on our ability to observe neural geometry. Without such geometry, no such structures can be discerned or understood.
In fact, the problem is deeper than geometry being unobservable. The problem is that in direct encodings, there is generally no means to represent ANN structure as a function of geometry and therefore no opportunity to exploit it at all. That is, a representation that simply evolves the weight of a connection between neurons A and B has no way to associate the weight of the connection with the relative locations of A and B when they have no location. Yet in any topographic map, the connectivity heavily depends on the geometric relationships among nodes in the map and nodes projecting afferent connections into the map. Thus, this lack of geometric structure has inadvertently handicapped NE from evolving brain-like structures. An NE method that endows evolved ANNs with an awareness of geometry might allow the ANNs to represent more biologically plausible structures.
In response to the limitations of direct encodings, other researchers have evolved indirectly encoded ANNs (Beer, 2000; Bentley & Kumar, 1999; Bongard, 2002; Dellaert & Beer, 1994; Gruau, Whitley, & Pyeatt, 1996; Hornby, 2004; Hornby & Pollack, 2002; Husbands, Smith, Jakobi, & Shea, 1998; Komosinski & Rotaru-Varga, 2001; McHale & Husbands, 2004; Stanley & Miikkulainen, 2003), which means that they evolve a compressed description of the ANN rather than the ANN itself. Such encodings can potentially describe geometric relationships within the ANN.
Building on this idea, this letter explores the implications of evolving networks that are aware of their geometry through the hypercube-based neuroevolution of augmenting topologies (HyperNEAT) method, which is uniquely sensitive to geometry. In HyperNEAT, the indirect encoding, called a compositional pattern producing network (CPPN), has the special property that it describes network connectivity as a function of neural geometry. The CPPN encoding is unique even among indirect encodings in that it explicitly assigns positional coordinates within the geometry to each node in the ANN, and this (x, y) coordinate of each node in the ANN is literally input into the CPPN. That way, encoded ANNs exhibit observable geometric regularities. The CPPN is able to encode an entire ANN by specifying, for any two neurons situated at locations A and B respectively, the connection weight wAB between them. Thus, neurons in HyperNEAT exist at defined coordinates in Euclidean space, enabling the geometric encoding of connectivity. When geometry is important to the problem, aligning the locations of neurons with the inherent geometry of the problem domain provides HyperNEAT a significant advantage over direct encodings, which have to learn the problem geometry one connection at a time.
The game of checkers is chosen as the domain for experimentation because checkers is intuitively geometric. The rules of checkers are relatively succinct, yet the strategy is potentially complex. A nice property is that the rules of movement for a piece do not change depending on the location of that piece on the board. Therefore, a representation that can extrapolate its strategies across the board to different locations gains a significant advantage over an approach that must learn the same concept for each square on the board individually.
To establish the role of geometry in effective learning, this letter examines the relative performance of HyperNEAT and another NE algorithm that learns from a predetermined network geometry intended to suit checkers. Both methods are compared to variants of a traditional approach that is completely unaware of geometry. ANNs are trained to defeat a fixed heuristic opponent. Thus, the goal in this work is not to produce the best possible checkers player but to analyze the properties of winning solutions by examining how they exploit geometry and the implications of such exploitation.
In particular, a major focus is the relationship between the geometry of solution ANNs and general performance against variants of the original heuristic that the winner was not trained against. To understand this relationship between geometry and generality, evolved receptive fields and influence maps (Hubel & Wiesel, 1962, 1968) are studied within the connectivities of general and less general players evolved by HyperNEAT. In addition, several checkers position evaluations are explored, demonstrating how subtle changes in the checkers board position result in changes in activation within different layers of the encoded ANN. This analysis reveals a surprising and novel insight: the networks that generalize most effectively display clear, smooth geometries in their connectivity patterns. In effect, they are smooth topographic maps optimized to evaluate checker boards. In contrast, less general networks (which still defeat the heuristic) exhibit jagged and irregular connectivity. The correlation between smoothness and generality can be exposed only through an NE algorithm that evolves networks with encoded geometry.
This correlation raises an intriguing insight about general behavior and its relation to geometry: an NE algorithm that does not encode structure as a function of geometry cannot be expected to exploit the same kinds of topographic correlations that are exploited by an indirect encoding and therefore cannot generalize in the same way.
Thus, a major result is that it is now possible to artificially evolve ANNs with topographic maps, moving them closer to biological plausibility, and potentially allowing artificially evolved specimens to begin to yield insights into neural function that previously were only the province of real biological study.
The letter begins with an overview of neuroevolution and the HyperNEAT method. Section 3 outlines the specific approach designed to learn regularities in checkers through HyperNEAT. The experiment is set up in section 4. Section 5 presents performance results, as well as an analysis of general and less general networks. This analysis is discussed in section 6, which also contains an outline of future work suggested by the main insights in this letter.
This section first reviews prior work in NE. Then the NEAT and HyperNEAT methods, which enable ANNs with geometry to evolve, are explained.
Neuroevolution is a field within evolutionary computation that focuses on training neural networks through evolutionary algorithms (Floreano et al., 2008; Yao, 1999). This approach applies the concepts of fitness, generations, populations, mutation, and sometimes crossover from evolutionary algorithms to evolve ANNs. It also benefits from the neural model, which is based on biology. In NE, the genotype represents an individual in the evolutionary algorithm that is transformed into an ANN during evaluation. After evaluation, the genotype receives a fitness that decides the parents of the next generation of individuals. NE can evolve any kind of ANN, including recurrent and adaptive networks (Floreano & Urzelai, 2000; Risi, Vanderbleek, Hughes, & Stanley, 2009; Soltoggio, Bullinaria, Mattiussi, Dürr, & Floreano, 2008). The way that the ANN is described by a genotype is called its encoding. This letter focuses on a specialized encoding that leverages geometry to create regular ANNs.
In early NE research, humans dictated the topology and encoding of evolved ANNs (Beer & Gallagher, 1992; Gomez & Miikkulainen, 1997; Kampfner & Conrad, 1983; Montana & Davis, 1989; Srinivas & Paranaik, 1991; Whitley, 1995; Wieland, 1991). While this approach allows human experts to design the topology and encoding with domain-specific optimizations, it is also limited by fixed topology. In contrast, evolving structure in addition to connection weights removes the burden of deciding the network topology from humans and places it on the learning algorithm (Angeline et al., 1993; Bongard, 2002; Fogel, 1992, 1993; Gruau, 1995; Hornby & Pollack, 2002; Lipson & Pollack, 2000; Miller, Todd, & Hegde, 1989; Pujol & Poli, 1998; Sims, 1994; Stanley & Miikkulainen, 2002b; Zhang & Muhlenbein, 1993).
The first methods to evolve both network structure and connection weights encoded networks directly, which means that a single gene in the genotype maps to a single connection in the network (Angeline et al., 1993; Pujol & Poli, 1998; Stanley & Miikkulainen, 2002b; Yao, 1999; Zhang & Muhlenbein, 1993). Although this approach is straightforward, it requires learning each connection weight individually. As a result, it is impossible to learn a regular pattern of connectivity without learning each connection in the pattern on its own. Again, human engineering is one approach to overcoming this limitation. For example, Togelius and Lucas (2005) introduced a symmetric ANN to power a symmetric robot, which reduced the amount of evaluations required by a factor of eight. Human engineering can capture patterns and regularities in the input and reduce them to a vector of numbers. However, ideally, evolution should be able to capture patterns and regularities on its own.
Indirect encodings give evolution the opportunity to explore patterns and regularities by encoding the genotype as a description that maps indirectly to the target structure (Bongard, 2002; Dellaert & Beer, 1994; Gruau et al., 1996; Hornby & Pollack, 2001; Kitano, 1990; Komosinski & Rotaru-Varga, 2001; Stanley, 2007; Stanley & Miikkulainen, 2003). That way, in neuroevolution, the genotype can be much smaller than the ANN, which results in fewer variables to optimize for the evolutionary algorithm. In fact, the CPPNs in HyperNEAT, described in section 2.3, are a geometric indirect encoding that draws inspiration from biology, capable of finding and exploiting geometric regularities in the problem domain.
However, it is possible to try to build some geometric grouping into the connectivity structure of an ANN even with a direct encoding. For example, an interesting attempt to integrate geometry into NE is Blondie24, an evolved checkers-playing ANN (Chellapilla & Fogel, 2001) that was able to reach expert-level play on a popular Internet checkers server and against an expert-level version of the program Chinook (Fogel, 2002). A similar approach is taken in Blondie25, a chess program that evolved neural networks to assist in evaluating the chess board (Fogel, Hays, Hahn, & Quon, 2004). The main idea in Blondie24 is that the ANN topology can be better engineered to respect the regularities inherent in the game. In particular, the weights of an ANN topology engineered by hand are evolved. Every subsquare (i.e., set of positions arranged in a square shape) of the board is input to a separate hidden node responsible for only that subsquare (see Figure 1). Connections are specified from the actual board inputs to their respective subsquares and also between the inputs and the final output node. The main idea in this engineered structure is that independent local relationships within each subsquare can be learned separately and then combined at a higher level in the network. Through coevolution (i.e., candidates were evaluated by playing against each other), Blondie24 was able to reach expert-level play on a popular Internet checkers server (Chellapilla & Fogel, 2001). However, an intriguing alternative would remove the need for engineering by learning geometric regularities on its own. This is the idea behind the HyperNEAT approach taken in this letter. The next section describes NEAT, which is extended later to implement HyperNEAT.
2.2. NeuroEvolution of Augmenting Topologies.
The approaches compared in this letter are variants of the neuroevolution of augmenting topologies (NEAT) method (Stanley & Miikkulainen, 2002b, 2004), which, like the approach in Blondie24 (Chellapilla & Fogel, 2001), evolves ANNs. In addition to evolving weights of connections, NEAT can build structure and add complexity. While the encoding in NEAT is direct, it turns out that it can be made indirect, which is the idea behind HyperNEAT. NEAT itself is a leading neuroevolution approach that has shown promise in board games and other challenging control and decision-making tasks (Aaltonen et al., 2009; Stanley & Miikkulainen, 2004; Stanley, Bryant, & Miikkulainen, 2005a; Stanley, Kohl, & Miikkulainen, 2005; Taylor, Whiteson, & Stone, 2006). This section reviews the NEAT method (for a full description see Stanley & Miikkulainen, 2002b, 2004; Stanley, Bryant, & Miikkulainen, 2005b).
NEAT is based on three key ideas. First, to allow network structures to increase in complexity over generations, a method is needed to keep track of which gene is which. Otherwise it is not clear in later generations which individual is compatible with which or how their genes should be combined to produce offspring. NEAT solves this problem by assigning a unique historical marking to every new piece of network structure. The historical marking is a number assigned to each gene based on its order of appearance over the course of evolution. The numbers are inherited during crossover unchanged and allow NEAT to perform crossover without the need for topological analysis. That way, networks of different organizations and sizes stay compatible throughout evolution.
Second, NEAT divides the population into species, so that individuals compete primarily within their own niches instead of with the population at large. In this way, topological innovations are protected and have time to optimize their structure before competing with other niches in the population. NEAT uses the historical markings on genes to determine to which species different individuals belong.
Third, NEAT begins with a uniform population of simple networks with no hidden nodes, differing only in their initial weights. Speciation protects new innovations, allowing diverse topologies to increase gradually in complexity over evolution. Thus, NEAT can start minimally and grow the necessary structure over generations. Through increasing complexity, high-level features can be established early in evolution and then elaborated and refined as new genes are added (Altenberg, 1994). Each component of NEAT was originally validated through a series of ablation studies (Stanley & Miikkulainen, 2002b). The most important concept for the purposes of this letter is that NEAT can evolve the right network structure and connection weights for the task. The next section reviews the extension of NEAT that allows it to evolve geometric relationships automatically.
2.3. CPPNs and HyperNEAT.
Like many other methods in machine learning, the reason that regular NEAT cannot explicitly learn geometric regularities is that when it learns to represent important local relationships (e.g., how a checkers piece in one square can be threatened by another in an adjacent square), it cannot extend that relationship as a pattern of connectivity across the entire neural structure connecting to the board. In other words, it needs to rediscover similar concepts multiple times.
The main idea in HyperNEAT is that it is possible to learn such relationships if the solution is represented indirectly, which means that it is a generative description of the connectivity of the ANN rather than embodying the connection weights of the ANN itself. As explained briefly in section 2.1, unlike a direct representation, in which every dimension in the solution space (i.e., the phenotype in evolutionary computation) is described individually (i.e., by its own gene), an indirect representation can describe a pattern of values in the solution space without explicitly enumerating every such value. That is, information is reused in such an indirect encoding, which is a major focus in the field of generative and developmental systems, the subfield of evolutionary computation from which HyperNEAT originates (Astor & Adami, 2000; Bentley & Kumar, 1999; Eggenberger, 1997; Hornby & Pollack, 2002; Lindenmayer, 1974; Mattiussi & Floreano, 2007; Stanley & Miikkulainen, 2003; Turing, 1952). 1
HyperNEAT is based on an indirect encoding called compositional pattern producing networks (CPPNs; Stanley, 2007). The idea behind CPPNs is that patterns such as those seen in nature can be described at a high level as a composition of functions that are chosen to represent several common motifs in patterns. For example, because the gaussian function is symmetric, when it is composed with any other function, the result is a symmetric pattern. The appeal of this encoding is that it allows patterns with regularities such as symmetry (e.g., with gaussians), repetition (e.g., with periodic functions such as sine), and repetition with variation (e.g., by summing periodic and aperiodic functions) to be represented as networks of simple functions, which means that NEAT can evolve CPPNs just as it evolves ANNs. While CPPNs are similar to ANNs, the distinction in terminology is particularly important for explicative purposes because in HyperNEAT, CPPNs describe ANNs. Formally, CPPNs produce a phenotype that is a function of n dimensions, where n is the number of dimensions in a geometric space. For each coordinate in that space, its level of expression is an output of the function that encodes the phenotype. Figure 2 shows how a two-dimensional pattern can be generated by a CPPN that takes two inputs.
The main idea in HyperNEAT is to extend CPPNs, which encode two-dimensional spatial patterns, to also represent connectivity patterns (D'Ambrosio & Stanley, 2007; Gauci & Stanley, 2007; Stanley et al., 2009). That way, NEAT can evolve CPPNs that represent ANNs with symmetries and regularities that are computed directly from the geometry of the task inputs. The key insight is that 2n-dimensional spatial patterns are isomorphic to connectivity patterns in n dimensions (i.e., in which the coordinate of each end point is specified by n parameters). Therefore, the connectivity patterns encoded by CPPNs can exhibit the same kinds of symmetries and regularities as those seen in CPPN-generated spatial patterns.
Consider a CPPN that takes four inputs labeled x1, y1, x2, and y2; this point in four-dimensional space can also denote the connection between the two-dimensional points (x1, y1) and (x2, y2), and the output of the CPPN for that input thereby represents the weight of that connection (see Figure 3). By querying every possible connection among a set of points in this manner, a CPPN can produce an ANN, wherein each queried point is the position of a neuron. Because the connection weights are produced as a function of their end points, the final structure is produced with knowledge of its geometry. In effect, the CPPN paints a pattern on the inside of a four-dimensional hypercube that is interpreted as an isomorphic connectivity pattern, which explains the origin of the name hypercube-based NEAT (HyperNEAT). The example in Figure 4 illustrates the natural connection between the function embodied by the CPPN and the geometry of the resultant network. Connectivity patterns produced by a CPPN in this way are called substrates so that they can be verbally distinguished from the CPPN itself, which has its own internal topology.
Recall that each queried point in the substrate is a node in an ANN. The experimenter defines both the location and role—hidden, input, or output—of each such node. As a rule of thumb, nodes are placed on the substrate to reflect the geometry of the task, which makes the setup straightforward (Clune, Ofria, & Pennock, 2008; D'Ambrosio & Stanley, 2007; Gauci & Stanley, 2007; Stanley et al., 2009). That way, the connectivity of the substrate becomes a direct function of the task structure.
For example, in a board game, the inputs can be placed on the substrate in a twodimensional plane just as their corresponding squares are arranged on the board, as in Figure 3. In this way, knowledge about the problem can be injected into the search, and HyperNEAT can exploit the regularities (e.g., adjacency, or symmetry) of a problem that are invisible to traditional encodings (see algorithm 1). (For full descriptions of HyperNEAT see D'Ambrosio & Stanley, 2007; Gauci & Stanley, 2007; and Stanley et al., 2009.) The next section explains in detail how checkers is represented and learned by HyperNEAT.
3. Approach: Learning Regularities in Checkers
The game of checkers is chosen for the experiments in this letter because it is intuitively geometric. While approaches like Blondie24 engineer geometry into the ANN topology in the hope that such engineering may be useful, the idea in HyperNEAT is to learn from geometry by generating the policy network as a direct function of task geometry. This section explains how that is done in the game of checkers.
To apply HyperNEAT to checkers, the substrate input layer is arranged in two dimensions to match the geometry of the checkers board (see Figure 5). Notice that the substrate in Figure 5 includes a hidden layer. Thus, it is analogous to two substrates (e.g., see Figure 3) stacked on top of each other. In particular, the two-dimensional input layer connects to an analogous two-dimensional hidden layer so that the hidden layer can learn to process localized geometric features. The hidden layer then connects to a single output node, whose role is to evaluate board positions. The CPPN distinguishes the set of connections between the inputs and the hidden layer from those between the hidden layer and the output node by querying the weights of each set of connections from a separate output on the CPPN (note the two outputs in the CPPN depiction in Figure 5). That way, the x and y positions of each node are sufficient to identify the queried connection, and the outputs differentiate one connection layer from the next. Because the CPPN can effectively compute connection weights as a function of the difference in positions of two nodes, it can easily map a repeating concept across the whole board.
In this way, the substrate is a board evaluation function. The function inputs a board position and outputs its value for black. To evaluate the board when it is white's turn to move, the color of the pieces can be reversed and then the sign of the result inverted. To decide which move to make, a minimax search algorithm runs to a fixed ply depth of four. Alpha-beta pruning (Knuth & Moore, 1975) and iterative deepening (Russell, Norvig, Canny, Malik, & Edwards, 1995) techniques increase performance without changing the output. The output of the substrate is the heuristic score for the minimax algorithm.
This approach allows HyperNEAT to discover geometric regularities on the board by expressing connection weights as a function of geometry. It is therefore unnecessary to manually engineer the network topology or divide the input space into subsections in an attempt to inject a priori theories about the key regularities in the game into the representation. Because HyperNEAT discovers geometric relationships on its own, an identical substrate can be applied to other board games even without knowledge of the game rules, contributing to the generality of the approach.
The experiment is designed to investigate the role of neural geometry in solving a problem that is clearly geometric. The idea is to learn to defeat a single fixed training opponent and then test for generalization against variations of this opponent. Thus, rather than producing the best possible checkers player, the aim is to analyze in detail the implications of a geometric representation, not only for learning but especially for generalization beyond what was trained.
The learned strategies are then tested against a nondeterministic variant of the same opponent. This variant has a 10% chance of choosing the second-highest-scoring move instead of the optimal move found in minimax search. This approach is similar to work done by Fogel (1993), who also implemented a percentage chance of picking a random move to diversify a deterministic opponent. Methods that evolve more general solutions should produce policies that win more such games.
The four compared approaches are chosen carefully to isolate the issue of geometric processing. Therefore, they are all variants of the same NEAT approach. This shared basis means that differences in performance are attributable to the way each approach processes its inputs. For all four approaches, input values of 0.5 and −0.5 encode black and white pieces, respectively. Kings are represented by a magnitude of 0.75, which is similar to the approach in Chellapilla and Fogel (2001), who showed that multiplying the standard piece input magnitude by 1.3 produces a good magnitude for kings in their approach. A single output expresses the value of the current board state for black.
Regular NEAT inputs a vector of length 32 in which each parameter represents a square on the board that can potentially hold a piece. NEAT evolves the topology and weights between the input and output nodes.
NEAT-EI is an attempt to enhance NEAT's ability to take into account geometric regularities across the board by supplying additional engineered inputs (EI). It has the same inputs as NEAT; however, the starting network topology is engineered as in Blondie24 to have additional inputs that focus on geometric regions of differing sizes (Chellapilla & Fogel, 2001; see Figure 1). The result of training NEAT-EI in this letter cannot be compared directly to Blondie24 because Blondie24 is the result of coevolution, while the policies in this letter are evolved against a fixed opponent. Rather than evolving the best possible player, the goal in this letter is to fairly compare the generalization of different representations, thereby isolating the issue of generalization.
HyperNEAT inputs are arranged in a two-dimensional 8 × 8 grid that forms the first layer of a three-layer substrate (see Figure 5). For HyperNEAT, NEAT evolves the CPPN that computes the connection weights of the substrate.
If it is indeed possible to exploit geometry to improve play, the better an approach can represent geometric relationships (through either learning or a priori engineering), the better that method should learn and generalize.
FT-NEAT (fixed topology NEAT) inputs are arranged in the same configuration as in the substrate in HyperNEAT (see Figure 5). FT-NEAT evolves the weights of this ANN but not the topology. Thus, FT-NEAT must evolve the connection weights of over 4000 directly encoded connections, helping to confirm that it is not just the particular topology of the substrate in Figure 5, but more important, the indirect encoding in HyperNEAT, that provides an advantage.
After the experimental comparison among the four methods, an extensive analysis of substrate visualizations from more and less general HyperNEAT-evolved players investigates how geometry influences generalization and the way evolved maps are organized.
4.1. Experimental Parameters.
Because HyperNEAT and NEAT-EI extend NEAT, both use the same parameters as NEAT (Stanley & Miikkulainen, 2002b). FT-NEAT also uses the same parameters, except it does not add nodes or connections. The population size was 120, and each run lasted 200 generations. The disjoint and excess node coefficients were both 2.0, and the weight difference coefficient was 1.0. The compatibility threshold was 6.0, and the compatibility modifier was 0.3. The target number of species was eight, and the drop-off age was 15. The survival threshold within a species is 20%. Offspring had a 3% chance of adding a node and a 5% chance of adding a link, and every link of a new offspring had an 80% chance of being mutated. Available CPPN activation functions were sigmoid, gaussian, sine, and linear functions. Recurrent connections within the CPPN were not enabled. Signed activation was used, resulting in a node output range of [ − 1, 1]. By convention, a connection is not expressed if the magnitude of its weight is below a minimal threshold of 0.2 (Gauci & Stanley, 2007); otherwise, it is scaled proportionally to the CPPN output. These parameters were found to be robust to variation in preliminary experimentation.
Performance in this section is measured in two ways. First, the fitness of each approach is tracked during training over generations, which gives a sense of relative training performance. Second, after training is complete, the best solutions from each run play 100 games against the randomized opponent, yielding generalization. The main question is whether HyperNEAT's ability to learn from geometry benefits its performance and generalization.
5.1. Training Performance.
Figure 6 shows the average generation champion fitness over evolution, averaged over 20 runs. While none of the runs of regular NEAT nor FT-NEAT was able to defeat the opponent within 200 generations, both HyperNEAT and NEAT-EI learned to defeat it in all runs. On average, it took NEAT-EI 57.85 generations to find a winning solution. HyperNEAT succeeds much more quickly, finding a winner in 8.2 generations on average. These differences are statistically significant according to Student's t-test (p < 0.05). This disparity highlights the critical importance of learning from geometry. While defeating the heuristic appears challenging with direct representations, it becomes easy if the solution is learned as a function of the board geometry.
Every generation champion that defeats the deterministic opponent plays 100 games against the randomized opponent. Because regular NEAT and FT-NEAT could never defeat this opponent, they are not included in this test. To make the comparison fair, only the most general solutions of each run are compared, which means the generation champion with the highest score computed by , where W and T are the number of wins and ties against the randomized opponent, respectively. The equation is used to convert a wins, losses, and ties metric to a single scalar score. That way, the generalization results focus on the best possible generalization for both methods when they learn to defeat an identical opponent. The best possible generalization represents what would result from an ideal validation of the trained opponents. While in the real world such idealized validation may not always be possible, assuming reasonable effort on the part of the experimenter, it is a yardstick for how well a system can be expected to perform in a reinforcement learning task. A similar approach to measuring generalization in such a task is taken by Gruau et al. (1996). Figure 7 shows the results of these solutions against the randomized opponent. HyperNEAT wins significantly more and loses significantly less than NEAT-EI. The geometric encoding allows HyperNEAT to generalize across the board.
5.3. Typical Solutions.
HyperNEAT's advantage is most evident in the middle game and later. As the game tree branches, deviation from the training opponent increases. Because HyperNEAT performs better in such novel situations, it is more general. For example, Figure 8 contrasts moves chosen by NEAT-EI solutions with those from HyperNEAT from the same unfamiliar position. NEAT-EI players unnecessarily sacrifice pieces, while HyperNEAT players rarely do from this position. Given that the evaluations during training consist of a single game against a deterministic opponent, the ability of a solution evolved during training to perform well in generalization tests against a nondeterministic opponent is significant. These typical solutions demonstrate the idea that because HyperNEAT evolves a pattern of weights across the geometry of the substrate, HyperNEAT is able to evolve a player that can both defeat the deterministic heuristic and simultaneously perform well in generalization tests, without any need for generalization pressure in the fitness function. Conversely, NEAT-EI struggles to generalize, suggesting that NEAT-EI learned a specific subset of board states instead of a general checkers strategy. In the case of NEAT-EI, generalization would likely benefit from playing additional games in a single evaluation.
The most general solution in all runs of NEAT-EI has 126 nodes and 1,106 connections. In contrast, the most general solution of HyperNEAT is a CPPN with only 23 nodes and 84 connections, which generates an ANN with 129 nodes and 3,979 connections. Figure 9 illustrates this dramatic compression afforded by indirect encoding in a typical HyperNEAT solution. In this way, HyperNEAT is able to explore a significantly smaller search space (i.e., CPPNs) while still creating complex structures (i.e., substrates).
5.4. Substrate Visualizations.
While the results so far establish that learning from geometry provides an advantage in both performance and generalization, an important question is exactly how this advantage is realized. This section aims to investigate this question by examining the internal connectivity and activation patterns of HyperNEAT-trained networks. It is important to note that this study of the topographic layout of nodes and connectivity within an evolved ANN is possible only because, unlike other neuroevolution algorithms (Angeline et al., 1993; Beer & Gallagher, 1992; Bongard, 2002; Braun & Weisbrod, 1993; Floreano et al., 2008; Fogel, 1993; Gomez & Miikkulainen, 1997; Gruau et al., 1996; Hornby & Pollack, 2002; Kitano, 1990; Koza & Rice, 1991; Lipson & Pollack, 2000; Montana & Davis, 1989; Moriarty & Miikkulainen, 1996; Opitz & Shavlik, 1997; Pujol & Poli, 1998; Sims, 1994; Srinivas & Paranaik, 1991; Stanley & Miikkulainen, 2002b; Yao, 1999; Zhang & Muhlenbein, 1993), the neurons within a HyperNEAT substrate are situated at geometric coordinates. This geometry is what affords the opportunity to observe patterns in their actual situated geometric context, giving insight into why such a context is important to learning in general and what kinds of opportunities it creates.
The particular focus of the analysis in this section is on the question of what kinds of connectivity patterns lead to generalization and what kinds do not. To investigate this difference, a group of four highly general HyperNEAT solutions and four HyperNEAT solutions that generalize less effectively (summarized in Table 1) are visualized in two different ways. First, the connectivity patterns of the solutions are visualized through images of influence maps and receptive fields. These images are arranged vertically within a single panel in one column (see Figure 10). The bottom image of each panel is a set of five influence maps that shows how individual inputs from the checkers board influence the entire hidden layer. The intensity at each position within each such map represents the magnitude of a single connection weight, and white triangles in the top-left corner of a position represent negative connection weights (the darker color denotes less influence). Thus, a full influence map shows all the weights projecting from a single input to the entire hidden layer. The five influence maps form a cross-shape, symbolizing that they represent images coming from five locations on the checkers board, as shown in Figure 10. Above the five influence maps are a similar five receptive field visualizations. These images are designed to show how each hidden node sees all of the inputs that can connect to it. Like the influence maps, the five receptive field visualizations are also shown in a cross—in this case, to represent where in the hidden layer the receiving node is located. The single image at the top is the receptive field of the single output node, which shows the connection weights from the hidden layer to the output, which is how the final computation of the board value is completed.
|General Solutions Against .||.||Less General Solutions Against .|
|Nondeterministic Heuristic .||.||Nondeterministic Heuristic .|
|Solution .||Wins .||Losses .||Ties .||.||Solution .||Wins .||Losses .||Ties .|
|General Solutions Against .||.||Less General Solutions Against .|
|Nondeterministic Heuristic .||.||Nondeterministic Heuristic .|
|Solution .||Wins .||Losses .||Ties .||.||Solution .||Wins .||Losses .||Ties .|
Notes: the two tables show the wins, losses, and ties of selected general and less general champions against the nondeterministic heuristic that are visualized later in this section. Note that the champions selected are not necessarily the champions of the run. Because the training phase involves only a single game against a deterministic heuristic, there is no explicit reward for generality in the fitness function. Even so, some of the runs produce solutions that generalize better than others. Note that because HyperNEAT generalizes well on average, the poorest generalizers from HyperNEAT still outgeneralize average NEAT-EI champions significantly. Nevertheless, the difference between these less general champions and those that are even more general still helps to elucidate the factors underlying effective generalization.
Second, visualizations of hidden-layer activation patterns for several board positions illustrate how boards are evaluated in the game tree (see Figure 11). Each figure with this type of visualization displays a row of activation patterns obtained by the champion of a particular run from Table 1. The activation pattern is illustrated by a column depicting the input board configuration (bottom), the hidden-layer node activation levels (middle), and the output activation (top), which is the value assigned to that board position for black.
These activation patterns are also organized topographically such that the activation level of each hidden neuron is depicted at that neuron's actual position in the substrate. Thus, it is possible to see how the activation levels relate to the network's geometry.
Each board position is an actual position encountered during alpha-beta search, so the overall visualization makes it possible to see how the network represents the difference between relatively good or bad situations. The board positions and activation patterns are ordered from left to right by increasing output value so that it is easy to see how increasingly good positions are represented by the network internally. Note that the sequence from left to right depicts board positions that were encountered during alpha-beta search rather than a sequence of moves during actual gameplay. The aim is to elucidate how judgments on board quality are represented.
The contrast between connectivity patterns from general (see Figure 12) and less general (see Figure 13) players is surprisingly revealing. In fact, the distinguishing characteristic of general play is visually apparent by simply observing its geometry: the connectivity patterns of networks that generalize most effectively exhibit smooth boundaries, while the boundaries of those that generalize poorly are jagged. This difference is particularly prominent in the influence maps (at the bottom of Figures 12 and 13), suggesting that influence maps from inputs reveal an important facet of geometry. These characteristics are consistent across all general and less general solutions in Figures 12 and 13. Thus, there is a strong correlation between generality and geometric smoothness.
The role of smoothness in generalization yields the important insight that the ability to represent smooth regularity is a critical prerequisite to consistent generalization. In contrast, jaggedness suggests memorization of the specific situations encountered while playing the particular opponent heuristic (recall that both general and less general solutions defeated the deterministic heuristic during training). The irregularity of the jagged solutions, which nevertheless beat the training heuristic, is an artifact of the peculiarities of the heuristic itself and not always useful when playing even slightly altered strategies.
Interestingly, smooth regularity is natural to represent only in a geometric context. After all, the connectivity that emanates from each input neuron in Figure 12 varies in a regular geometric fashion as the source neuron shifts position across the board (observe the pattern at different locations within each cross). Only an indirect encoding that describes connectivity as a function of geometry is likely to consistently yield such regularities. A direct encoding cannot describe how a pattern varies smoothly over space. Therefore, in a direct encoding, each individual connection is learned separately, most likely yielding jagged patterns. (Note that because direct encodings do not have a geometry, they cannot be visualized in this way; however, it is exactly this fact that prevents them from expressing smooth patterns that vary across geometry.) In fact, if the patterns yielded by direct encodings could be situated geometrically, they would likely be significantly more irregular than even those in Figure 13, which still are at least the product of indirect encoding.
In effect, the patterns in Figure 12 are evolved topographic maps for the game of checkers. A special (and unique) kind of receptive field is evolved in each case that moves in a predictable regular fashion across the hidden layer in accordance with the position of the source neuron from the input layer. In the less general solutions (see Figure 13), these maps are less regular and more distorted, hurting generalization.
Another important observation about the general patterns in Figure 12 is that while they are all regular and smooth, they are also all different from each other. That is, the receptive field structure in Figure 12a is unlike Figures 12b, 12c, and 12d; Figure 12b is unlike Figures 12a, 12c, and 12d, and so on. Therefore, interestingly, the implication is that as long as smoothness and regularity are achieved in the influence maps, there are multiple ways to solve the same problem effectively, perhaps explaining why HyperNEAT beats the heuristic so quickly while still generalizing often. It is the bias toward smooth topographic maps that portends the ability to generalize.
As explained in Figure 11, Figures 14 through 21 show how these connectivity patterns, both general and less general, integrate to evaluate actual board positions encountered in the game tree. It is important to note that the precise value of the output node activation (shown at the top of each panel) is not important; because evaluations are performed within an alpha-beta search, only the relative output affects decision making. For example, a negative output value does not necessarily indicate a poor evaluation, and vice versa. It is also important to note that differences in board positions are often reflected in subtle changes in neural activation among hidden nodes. Thus, it is occasionally difficult to perceive these changes visually. Nevertheless, they are often perceptible by comparing activation patterns closely, and their subtlety signifies the precision with which the substrate disambiguates similar board positions.
Through Figures 14 to 21, it is once again clear that there are many ways to solve checkers against the heuristic. However, it is also apparent that the overall activation patterns on the hidden layer exhibit definite shapes that are tied inextricably to the geometry of the board itself. That is, areas of high activation are normally all adjacent and separated from areas of low activation, even if these areas appear in different locations for different solutions. Thus, the overall activation patterns, which are realized through the individual neural connectivity patterns in Figure 12, are also fundamentally geometric, combining the joint assessments of each individual receptive field.
Less general solutions (see Figures 18–21) yield activation patterns that are mainly jagged and discontinuous. These shapes do not resemble general solutions (see Figures 14–17), although they still exhibit patterns that are geometric. The jaggedness in the patterns suggests that the solutions are able to defeat the deterministic heuristic by memorizing certain states, and not by encoding a holistic pattern that describes the dynamics of checkers.
The general solutions (see Figures 14–17) typically favor a holistic strategy. For example, in Figures 14 and 15, keeping pieces in a tightly bound group at the back of the board is rewarded, although the lateral focus of density (left versus right) differs. In contrast, the substrates in Figures 15 and 16 favor solutions that are more aggressive and attempt to control of the center of the board. Nevertheless, the principle that unifies all these approaches is their generality; they are sensitive to relative concentrations of groupings of pieces.
Less general solutions (see Figures 18–21), while often reasonable, exhibit idiosyncratic holes in their approach that are reflected in their more piecemeal activation patterns. As described in Figures 18 to 21, such idiosyncrasies often yield specific evaluations that are fundamentally flawed. For example, position 19d is rated relatively highly yet leaves black open to a double-jump on the next turn, after black takes white's piece. Position 21e rewards black for advancing up the board, but does not account for the fact that white will capture the leading piece on the next turn. The fact that solutions that do not have a smooth geometry make such mistakes despite defeating the deterministic heuristic further suggests that generality is linked to smooth geometry.
The analysis in this section shows what it means to learn from geometry. In effect, learning from geometry means being able to correlate topographic maps to the geometry of the world. This ability affords smooth regular connectivity patterns, which this section showed are often correlated to the more general checkers players. The next section explores the deeper implications of this discovery and what it means for artificial evolution in general.
6. Discussion and Future Work
A major difference between traditional multilayer perceptrons (MLPs; McClelland, Rumelhart, & Hinton, 1986) and biological brains is that real brains profusely exploit topographic relationships (Sporns, 2002). Some artificial neural models such as self-organizing maps (SOMS; Kohonen, 1981) and cortical models (Bednar and Miikkulainen, 2006; Bednar et al., 2002; Swindale, 1996) exhibit geometric structure, but the connectivity and topographic correlations in such models are not evolved. A key contribution of this letter is to show that it is possible for an evolutionary algorithm to actually evolve its own topographic maps that are domain appropriate. This development is intriguing because it means that neuroevolution algorithms can now produce structures more reminiscent of biological brains.
Furthermore, an important result is to show why evolving such structures is advantageous. In particular, at least in checkers, visualizing artificially evolved influence maps and receptive fields suggests an intimate connection between generalization and geometry. The connectivity patterns that exhibit smoothness, a qualitative assessment of the gradient across the hypercube of connection weights, were shown to be correlated to generalization, a quantitative assessment of the substrate against new opponents.
HyperNEAT is biased toward creating general players because the low complexity of the initial population of CPPNs tends to start evolution with simple, smooth geometries. However, it is not guaranteed to produce general results; several runs yielded less general solutions. Nevertheless, it is important to note that even these less general solutions still generalize significantly better than NEAT-EI on average, suggesting that NEAT-EI cannot easily encode the same smooth regularities demonstrated in HyperNEAT. Even worse, no runs of regular NEAT or FT-NEAT were able to defeat the deterministic heuristic in training, illustrating the necessity of capturing at least some geometric regularities, whether through an indirect encoding such as HyperNEAT or an engineered topology such as NEAT-EI. While engineering geometry into the network connectivity (as with NEAT-EI) provides NEAT a necessary advantage, it is not able to outperform HyperNEAT's ability to learn from geometry.
HyperNEAT solutions generalize significantly better than NEAT-EI solutions even though both methods trained against (and eventually defeated) the same heuristic in training. This difference is explained by HyperNEAT's indirect encoding: because HyperNEAT CPPNs initially are much smaller than NEAT-EI genomes, they are biased toward representing substrates that are highly regular but also successful in the domain of checkers. Because the direct encoding must learn each link weight individually, it searches through a comparatively high-dimensional space of neural networks, while the indirect encoding searches through a compressed (and hence lower-dimensional) space of solutions by leveraging its more powerful representation. HyperNEAT's representation naturally describes the geometric regularities of the problem domain. This capability helps in checkers because the domain (like many others) is inherently geometric. For example, the same rules generally apply to each piece at any position. Thus, the domain of checkers helps to illustrate the advantage of an indirect encoding based on geometry such as in HyperNEAT.
However, the scope of domains that are inherently geometric is not limited to checkers and other board games. For example, Clune et al. (2008; Clune, Beckmann, Ofria, & Pennock, 2009; Clune, Ofria, & Pennock, 2009); D'Ambrosio and Stanley (2007, 2008), Gauci and Stanley (2007), and Stanley et al. (2009) show that visual discrimination and robot control domains can also benefit from indirect encoding through geometry. The inspirations for such domains are the vision and control systems of the human brain. In fact, topographic maps, which often have a geometry isomorphic to the external environment, are studied in the context of biological brains (Chklovskii & Koulakov, 2004; Churchland, 1986; Goodhill & Carreira-Perpinn, 2002; Kandel, Schwartz, & Jessell, 2000; Sporns, 2002). For example, the somatotopic representation of the human body in the brain exhibits a similar geometry to the body itself (Nakamura et al., 1998; Yang, Gallen, Schwartz, & Bloom, 1993). Interestingly, ANNs evolved with HyperNEAT have receptive fields and influence maps that can be visualized much like such topographic maps in biological brains. Thus, topographic maps in ANNs evolved by HyperNEAT are reminiscent of their more sophisticated biological counterparts, suggesting the start of an intriguing new direction of research in artificially evolved substrates. In addition, not only do such maps arise, but in this letter, their analysis helped to establish a connection between the smoothness and regularity of the geometry of such a map and its generalization. As a result, the surprising insight is that a qualitative assessment of evolved topographic maps translates directly to quantitative performance results in the generalization test.
Interestingly, any domain that exists in a space of multiple dimensions contains at least an implicit geometry that can be potentially exploited through an indirect encoding based on geometry. Clune et al. (2008) demonstrate that even when the geometry of an ANN that controls a robot is scrambled, HyperNEAT is able to find regularities within the scrambled geometry. Thus, future work for this approach will explore other challenging domains. The board game Go is one such appealing candidate for further research because its geometry is similar to that of checkers. Beyond board games, promising avenues include continuing work in vision (Gauci & Stanley, 2007) and robot control (D'Ambrosio & Stanley, 2007). While early such work focused on relatively simple problems, it is not known how close evolved indirect encodings can approach the complexity of biological brains, which are clearly suited for such tasks. Even if approaches such as HyperNEAT do not reach such ambitious scale, lessons learned along the way, such as the connection between smooth geometry and generalization, promise to be illuminating.
For example, an interesting question is whether ANNs evolved by HyperNEAT for visual tasks might resemble features in V1 or other parts of the biological visual processing hierarchy (Bednar & Miikkulainen, 2006; Hubel, 1988). While the human primary visual cortex contains about 140 million neurons (Leuba & Kraftsik, 1994), HyperNEAT has evolved functional networks with millions of connections (Stanley et al., 2009). Furthermore, while biological brains (including the visual cortex) exhibit synaptic plasticity (Hubel, Wiesel, & LeVay, 1977), ANNs with plastic synapses have been evolved in the past (Floreano & Urzelai, 2000, 2001; Floreano & Mondada, 1996; Risi et al., 2009; Soltoggio et al., 2008), and in principle, HyperNEAT can potentially evolve the geometry of the learning rules, taking it another step closer to biological plausibility. That is, HyperNEAT can potentially assign plasticity roles to connections in a geometric pattern, which is necessary if plastic structures with millions of connections are to be evolved. Thus, while the evolved maps in this letter are static, in principle the capability to encode such maps suggests that evolving plastic maps with similar properties (e.g., geometry) is plausible.
Stepping back to the role of geometry in learning algorithms today, the results and analysis in this letter suggest that an important prerequisite to exploiting geometry in learning is to be aware of it. That the CPPNs in HyperNEAT literally see the positions of the nodes being connected affords the ability to exploit the domain geometry by creating smooth, semiregular patterns. To date, this ability to see the geometry of the substrate is unique, yet it portends the importance of endowing future algorithms with a similar capability if they are to exploit domain geometry effectively. Simply imagining trying to learn tic tac toe, a simple game, on a scrambled board illustrates the urgency of this consideration. Once the capability to perceive geometry is made available, an exciting new research direction with interesting biologically parallels opens up.
This letter argued that representing evolved ANNs as indirect functions of their geometry evolves structures that are closer to structures seen in biological brains than those evolved by prior NE approaches. In addition, such a method is able to exploit the underlying geometric regularities in a problem to quickly find elegant solutions to complex problems, provided that geometry plays a role in the problem domain.
The role of geometry was shown to be potentially useful to machine learning performance in the domain of checkers. Regular NEAT and FT-NEAT were not able to defeat the deterministic heuristic in a single run of training, while NEAT-EI and HyperNEAT were able to defeat the heuristic in all 20 runs. HyperNEAT was able to find solutions relatively quickly by searching through the low-dimensional space of CPPNs, while NEAT-EI took significantly longer, searching through the high-dimensional space of ANNs. In addition, solutions produced by HyperNEAT generalized significantly better than solutions produced by NEAT-EI, suggesting a link between HyperNEAT's perception of geometry and generality.
This link was confirmed through a visual study of general and less general HyperNEAT solutions and their performance in training. A correlation was drawn between the smoothness and continuity of connectivity patterns across the layers of solutions and their generalization performance, suggesting that general solutions encode ANNs that are smooth and regular, while less general solutions encode ANNs that are jagged and discontinuous. The CPPNs that encoded jagged ANNs were specialized for the specific games of checkers seen in training, while the CPPNs that encoded smooth ANNs were more general checkers players.
These results suggest that NE methods should ideally see the geometry of the domain and be able to encode and represent geometry in a way that creates smooth and regular ANNs. In this way, the ANNs produced by NE can more closely resemble neural networks seen in natural brains. In the future, the results of such artificial evolutionary approaches will offer increasing relevance to researchers beyond the field of NE.
Special thanks to Martin Fierz for providing the Simplech checkers engine. Special thanks also to our anonymous reviewers for helping to improve this work significantly.
The ideas in GDS extend back to Turing (1952), who experimented with pattern formation through reaction-diffusion systems.