Studying coevolutionary systems in the context of simplified models (i.e., games with pairwise interactions between coevolving solutions modeled as self plays) remains an open challenge since the rich underlying structures associated with pairwise-comparison-based fitness measures are often not taken fully into account. Although cyclic dynamics have been demonstrated in several contexts (such as intransitivity in coevolutionary problems), there is no complete characterization of cycle structures and their effects on coevolutionary search. We develop a new framework to address this issue. At the core of our approach is the directed graph (digraph) representation of coevolutionary problems that fully captures structures in the relations between candidate solutions. Coevolutionary processes are modeled as a specific type of Markov chains—random walks on digraphs. Using this framework, we show that coevolutionary problems admit a qualitative characterization: a coevolutionary problem is either solvable (there is a subset of solutions that dominates the remaining candidate solutions) or not. This has an implication on coevolutionary search. We further develop our framework that provides the means to construct quantitative tools for analysis of coevolutionary processes and demonstrate their applications through case studies. We show that coevolution of solvable problems corresponds to an absorbing Markov chain for which we can compute the expected hitting time of the absorbing class. Otherwise, coevolution will cycle indefinitely and the quantity of interest will be the limiting invariant distribution of the Markov chain. We also provide an index for characterizing complexity in coevolutionary problems and show how they can be generated in a controlled manner.
In this study we focus on coevolutionary systems that are specified by the: (1) coevolutionary problem—the set of all alternatives (e.g., strategies in a two-player game) and a pairwise preference relation indicating the superior alternative (e.g., whether when playing the game one strategy defeats another one), and (2) coevolutionary process—the population of alternatives undergoing a process of selection and variation that is guided only by their interaction outcomes. Coevolutionary systems are often used in competitive settings where the lack of absolute fitness measures necessitates that one exploit the strategic nature of interactions in coevolving populations. In this context, coevolutionary systems provide a natural alternative for solving test-based problems (Hillis, 1990; de Jong and Pollack, 2004) that include two-player strategic games whereby opponent strategies provide test cases for fitness evaluations (Runarsson and Lucas, 2005). In turn, game-theoretic-based frameworks have been developed to provide tools for analysis of coevolutionary systems (Ficici et al., 2005; Tiňo et al., 2013).
The use of relative and dynamic fitness evaluations in coevolutionary systems introduces unique challenges that make these systems difficult to analyze and understand in comparison to evolutionary systems. They are collectively referred to as coevolutionary pathologies. In the classic example of two-population predator-prey coevolution, the Red Queen effect is coined to describe the interdependence in the fitness evaluations of members in one population to the other even though there is no exchange of genetic materials (Cliff and Miller, 1995; Janssen et al., 2016). Disengagement occurs when there are large gaps in competence levels between the populations, leading to a loss of selection pressure, decoupling of the coevolutionary system, and then onset of random drift in the populations (Cartlidge and Bullock, 2004). For problems with intransitivity in the relationships between the solutions, cyclic dynamics have been observed whereby the population overspecializes to copies of a solution that is exploited by another solution later in the coevolutionary process (Darwen and Yao, 1997). In the more extreme case, coevolution forgets a solution it has discovered earlier—the solution that is driven to extinction is adaptively found later (Rosin and Belew, 1997).
Although there are broader implications in design of effective coevolutionary systems, pathologies are usually associated with the selection part of the coevolutionary process, particularly on the use of pairwise-comparison-based fitness measures. Most studies would frame the pathologies in specific settings for which relevant methodologies can be used for analysis. These include measurements for monitoring the progress of coevolutionary systems using generational samples of test cases (Cliff and Miller, 1995; Stanley and Miikkulainen, 2002; Bader-Natal and Pollack, 2005) and those where one has a global view and can objectively measure performance of solutions (Floreano and Nolfi, 1997; Chong et al., 2012). In Ficici et al. (2005) and Tiňo et al. (2013), a formal approach is taken to study coevolutionary dynamics. Under the usual evolutionary game theoretic setting such as the use of an infinite population, a full characterization is given on coevolutionary dynamics (as changes to the ratio of individuals adopting a pure strategy) for various dynamical maps modeling different selection operations.
Here, we focus on coevolutionary systems that use pairwise-comparison-based fitness measures. However, it remains an open challenge to study this restricted class of coevolutionary system since the rich underlying structures associated with pairwise-comparison-based fitness measures are often not taken fully into account. Although cyclic dynamics have been shown in coevolutionary systems, most past studies such as Samothrakis et al. (2013) would focus on short cycles (of length three). Longer length cycles can occur in coevolution. Crucially, there is no complete characterization of cycle structures and their effects on coevolution. Our motivation is to develop a new framework to address this issue formally. Our approach uses the digraph representation for coevolutionary problems to capture the full structures in the binary relations between candidate solutions. Coevolutionary processes are modeled naturally as discrete time Markov chains operating on discrete state spaces (the sets of vertices of digraphs). In this manner, different coevolutionary algorithms would correspond to different implementations of random walks on digraphs.
We first show that coevolutionary problems admit a qualitative characterization: a coevolutionary problem is either solvable (the digraph is reducible with a subset of solutions that dominates the remaining solutions of the solution space) or not. This has an implication on coevolutionary search where we further develop the framework to construct quantitative tools for appropriate analysis and demonstrate their applications through case studies. We will show that coevolution of solvable problems corresponds to an absorbing Markov chain for which we can compute the expected hitting time of the absorbing class. Otherwise, coevolution will cycle indefinitely and the quantity of interest will be the limiting invariant distribution of the Markov chain. We also suggest an index for characterizing complexity in coevolutionary problems and show how a hierarchy of increasingly complex coevolutionary problems can be generated in a controlled manner.
This article is organized as follows. In Section 2, we formulate the digraph representation and provide several results characterizing cycle structures in coevolutionary problems. Section 3 formulates coevolutionary processes as random walks on digraphs and provides examples how specific Markov chains are constructed from coevolutionary algorithms operating on digraphs. Section 4 introduces relevant quantitative characterizations for coevolutionary systems (expected hitting times and stationary distributions of Markov chains). These quantitative tools will be demonstrated through theoretical and computational case studies. Section 5 will discuss the main findings and wider implications of our framework with remarks for future studies.
2 Digraph Representation of Coevolutionary Problems
We first present basic definitions and facts on digraphs relevant to formulating our framework to make this article self-contained. Our notation and terminology follow standard digraph theory texts and readers can refer to Bang-Jensen and Gutin (2009) for detailed expositions of the following. Our focus is to show how digraphs provide appropriate mathematical structures that fully capture the pairwise preference relations between objects of interest that one would encounter in typical coevolutionary problems such as games. Later, we would exploit classic results in digraph theory to provide a complete characterization of cycle structures in coevolutionary problems.
2.1 Digraphs—Definition, Notation, and Terminology
A digraph consists of two sets: (i) a non-empty, finite set of elements called vertices, and (ii) a finite set of elements corresponding to ordered pairs of distinct vertices, , called arcs (or directed edges). The arc set is a binary relation on the vertex set ( is a subset of the Cartesian product of with itself, ).
The order of a digraph is given by the number of vertices (often denoted as ). The size of is given by the number of arcs.
A graph consists of: (i) a non-empty, finite vertex set , and (ii) a finite set corresponding to unordered pairs of distinct vertices, , called edges. The underlying graph of a digraph is the graph with the same vertex set and the corresponding edge set whereby arcs are replaced by edges with no parallel edges and loops.
We often use a simpler notation and denote the arc as . The tail and head are the end-vertices of . Adjacency between end-vertices and implies an arc or . As in Bang-Jensen and Gutin (2009), we consider the digraph without parallel arcs between adjacent end-vertices (arcs having the same tail and same head) and loops (the arc where the tail and head is the same vertex).1
The definition of digraphs and in particular the binary relation on the vertex set that can be constructed allows for rich structures to be captured. We will consider notions of orientation, biorientation, complete, and semicomplete.
The orientation of a graph is the digraph where each edge corresponding to a pair of adjacent vertices is connected by an arc (either or only). The biorientation of a graph is the digraph where each edge in the graph is replaced by the arc or or both arcs and .
Obviously, any digraph is a biorientation. An oriented graph is a specific biorientation since any pair of adjacent vertices is connected by an arc only. In a complete graph, every pair of distinct vertices is adjacent.
A semicomplete digraph is a biorientation of a complete graph (since not all pairs of distinct vertices are completely bioriented, in which case they are only oriented). A tournament is a complete oriented graph.
Tournaments form a special and well-studied class of semicomplete digraphs (Bang-Jensen and Gutin, 2009). Tournaments and their generalizations (semicomplete digraphs) would be the focus later in this article.
2.2 Neighbourhoods and Degrees
We note results from the set difference between and in the usual manner. The vertices in , , and are called out-neighbours, in-neighbours, and neighbours, respectively (Bang-Jensen and Gutin, 2009).
A related notion of neighbourhoods is the degrees in a digraph.
2.3 Walks, Paths, and Cycles
Consider a digraph , whereby vertices and arcs from are labeled in a manner such that the tail and head of is and , respectively, for every (Bang-Jensen and Gutin, 2009).
A -walk in the digraph from to is the alternating sequence of vertices and arcs . The set of vertices of the walk is while the set of arcs of the walk is . is said to be closed if , otherwise it is open. The length of a walk is the number of arcs in it and by definition of is simply .
A -path is a walk where all the vertices in it are distinct. A k-cycle is a closed -walk of length on the -path.2
We usually denote , , and without the arcs, , since is defined in the context of (Bang-Jensen and Gutin, 2009). and on are by definition subdigraphs of , e.g., , , and .
The path is said to be Hamilton if is the same as . A Hamilton cycle has . A digraph is said to be Hamiltonian if it contains a Hamilton cycle.
2.4 Classes of Coevolutionary Problems as Digraphs
We state how coevolutionary problems can be represented as digraphs in the following.
A coevolutionary problem is defined as an ordered pair where represents the solution set and is the pairwise preference relation on . This is represented as the digraph . The vertex set is the solution set (each distinct vertex corresponds to a single unique solution). The arc set is the binary relation on , (each biorientation corresponds to the pairwise preference relationship between each pair of solutions).
The underlying graph is complete since there is a relationship between every pair of solutions in a coevolutionary problem. Each problem is specified by the number of unique solutions and associated preference relation on those solutions, which is described by the biorientations on . We use the term coevolutionary digraph to emphasize the digraph representation of a coevolutionary problem. It should be clear from the context of subsequent presentations whether we are referring to coevolutionary problems or just the digraphs as mathematical structures.
For this study, we consider two general classes of coevolutionary problems with their corresponding digraph representations.
Consider the coevolutionary problem where is the solution set with . Its underlying graph is complete. Then: (i) a coevolutionary tournament is an orientation of , and (ii) a coevolutionary semicomplete digraph is a biorientation of . For each , let and be the sets of all tournaments and semicomplete digraphs of order , respectively. and represent the family of all coevolutionary tournaments and semicomplete digraphs with candidate solutions, respectively.3 generalizes since adjacent vertices in can be completely bioriented. That is, .
To provide a direct context, there is a wide range of coevolutionary problems that can be modeled as two-player strategic games. In each case, the game consists of a set of pure strategies and a dominance relation on those strategies. Both players have access to the same strategy set. Each player will select a single pure strategy from the strategy set and then implement that strategy during game-play. The game outcome is decided by the dominance relation on the two pure strategies implemented for the game-play. Some games have only win-lose game outcomes—the winning strategy is said to dominate the losing strategy. Other games (two-player board games such as chess) have a win, lose, or draw outcome. Digraphs naturally represent these coevolutionary problems. The set of vertices of the digraph is the strategy set. The dominance relation is captured and described by the set of arcs. We do not consider game-plays between copies of the same pure strategy for simplicity—there is no self-loop in the digraph.
For any pair of strategies , the winning strategydominates a losing strategy. This relation is represented by the orientation (or if we say is dominated by ).4 A draw is represented by the complete biorientation .
The notion of domination can be extended to subsets of (a vertex is a subset of with a single element). Consider two disjoint subsets of , and , whereby . denotes that all vertices dominate all vertices . We say is the dominant subset with respect to . Conversely, . The tournament provides a representation for all strategic games with win-lose game outcomes. Semicomplete digraph as a further generalization of the tournament can be used to represent any strategic game with win, lose, or draw outcomes.5
2.5 Structures and Higher Organizations in Coevolutionary Digraphs
In practice, one often is confronted with a very large coevolutionary digraph with little (connectivity) information. A coevolutionary tournament with candidate solutions has pairs of comparisons. This exponential increase in pairwise comparisons is further compounded by a high (and possibly exponential) growth rate in the number of parametric representations of candidate solutions. Consider a simple normal form game such as the iterated prisoner's dilemma restricted to just memory-one, reactive (self and opponent's past plays) strategies. If one uses a direct look-up table representation, a -choice game would generate different combinations of strategy tables (Chong et al., 2012). A two-choice game would generate eight strategy tables and 784 pairwise comparisons, which increase to at least strategy tables and pairwise comparisons for a six-choice game in a full tournament.
Performing a brute-force computation on a very large coevolutionary digraph (e.g., decide if the problem has a single dominant solution) quickly becomes a very expensive task. Instead, we study specific structures leading to some form of higher organizations in coevolutionary digraphs that can provide useful characterizations of coevolutionary problems. These structures emerge from constraints that could be placed on the biorientations of coevolutionary digraphs.
2.6 Connectivity Structures in Digraphs
We first define basic connectivity structures in digraphs before introducing qualitative characterizations of tournaments having specific connectivity conditions.
A digraph is said to be strongly connected (or strong) if for each pair of distinct vertices there exist both -path and -path.
Given a pair of distinct vertices , is said to be reachable from if and only if there is an -path in . A vertex is reachable from itself and so by definition a digraph with a single vertex is strong. A strong digraph will have each vertex reachable from all other vertices. There exists a closed Hamilton walk that cover all vertices in the digraph. The following describes strong digraphs having specific cycle structures.
A digraph of order that has -cycles of all lengths is said to be pancyclic. The digraph is further said to be vertex-pancyclic if for every and every , there is a -cycle containing .
Acyclic digraphs are those without cycles and are defined formally as follows.
Let be an ordering of . If is such that for all arcs , , the digraph is said to have an acyclic ordering. Every acyclic digraph has a vertex with zero in-degree and a vertex of zero out-degree .
Every acyclic digraph has an acyclic ordering (Bang-Jensen and Gutin, 2009).
A digraph is transitive if for every pair of distinct arcs , and .
A tournament is transitive if and only if it is acyclic (Bang-Jensen and Gutin, 2009).
For each , let be the set of all tournaments of order . Vertices for each tournament can be labeled (indexed) in a manner so that is ordered according to . This is done using the score sequence.
The score sequence of a tournament is the -tuple with the constraints . Each , , corresponds to the number of vertices that dominates, i.e., .
The following two definitions (Moon, 1968) are crucial in the next subsection where we present our results on how vertices in a digraph can be grouped or classified in some manner that reveals useful structures.
Consider a vertex partition of vertices of a tournament into two disjoint, nonempty subsets and whereby and . is reducible if it can be vertex-partitioned such that ( is the dominant subset). Otherwise, it is irreducible.
Some reducible digraphs further admit vertex partition (decomposition) into multiple, disjoint subdigraphs, which is defined as follows.
A digraph can be decomposed into subdigraphs where and when .
2.7 Qualitative Characterizations of Coevolutionary Digraphs
We first summarize several classic results of tournament theory on connectivity structures in tournaments. The results will be presented in the original form of the (referenced) primary texts (slightly restated for consistency with our notations).
A tournament is strong if and only if it is irreducible.
Every strong tournament is Hamiltonian.
We exploit these theoretical results and provide useful qualitative characterizations to the family of coevolutionary tournaments . In this article, we state and show that a coevolutionary problem is either solvable (its corresponding tournament is reducible) or not (the tournament is irreducible).
A coevolutionary tournament on is either reducible or irreducible.
We next show that a solvable coevolutionary problem has a subset of solutions that are preferred over the remaining solutions.
A reducible coevolutionary tournament of order can be vertex-partitioned into two disjoint nonempty subsets and . The dominant subset contains solutions that dominate all remaining solutions in the other subset . The minimal dominant subset contains a single solution.
The first part of the lemma is simply a direct consequence of the definition of reducibility of tournaments. It remains to prove the second part. We first make note that every tournament can be decomposed into subtournaments whereby (Moon, 1968):
Every vertex in dominates every vertex in if .
Every subtournament is either irreducible or transitive.
No two consecutive subtournaments and are both transitive.
Consider the trivial case whereby the tournament is transitive, . As a consequence of Theorem 23, a tournament of order is transitive if and only if it is acyclic. Obviously, we can always vertex partition a transitive tournament by taking the vertex with in-degree as and all the remaining vertices as so that . For a reducible tournament, assume that it can be decomposed into two subtournaments (). Since the two subtournaments and cannot be both transitive, then one of them must be irreducible. Let . with a single vertex is by definition strong and as such irreducible.
We need to show the case above only where the dominant subset is minimal. There are other cases as well; for example, is transitive and is irreducible. Here, even though , the single dominant vertex in a transitive obviously dominates all other vertices in and as well. In another case, we have both irreducible subtournaments. For these coevolutionary tournaments, the maximal dominant subset that induces and for which is of order .
For coevolutionary problems that are not solvable, we have the following result.
An irreducible coevolutionary tournament is strongly connected. It is Hamiltonian and furthermore, vertex-pancyclic.
The main implication is that an irreducible coevolutionary tournament does not have a subset of dominant solutions. In addition to every contained in a 3-cycle, this coevolutionary tournament is vertex-pancyclic. One can find the solution that dominates all other solutions but is dominated by , which in turn is dominated by (see Figure 1). This is a consequence of the Hamilton cycle in the tournament. In the context of a game and optimality of a strategy that is defined in terms of the number of opponent strategies it defeats, one finds the situation whereby the best strategy defeats all but the worst strategy. The lack of dominant solutions becomes more apparent in the case of regular coevolutionary tournaments whereby each solution has the same score with , which generalize with (as in the game rock–paper–scissors).
Our results can be extended to the family of coevolutionary semicomplete digraphs as well. The arguments behind Theorems 21, 22, and 23 for tournaments can also be made for their simple generalization in the form of semicomplete digraphs. For example, Theorem 23 on strong connectivity and vertex pancyclicity for tournaments has been restated for semicomplete digraphs in Bang-Jensen et al. (1997). The original argument (Moon, 1968) that makes use of a proof by induction of a -cycle (where containing a vertex and the existence of a -cycle containing remains unchanged if one replaces instances of tournaments with semicomplete digraphs. One can make an observation (e.g., see Figures 1a and 1b) that the introduction of 2-cycles does not make a semicomplete digraph vertex pancyclic. Instead, it arises from the presence of 3-cycles and the Hamilton cycle.
Unlike the case of irreducible coevolutionary digraphs, one can employ search procedures to discover the dominant subset in a reducible coevolutionary digraph. We first introduce the following notion on strong connectivity of subdigraphs of (Bang-Jensen and Gutin, 2009). Then, we conclude with a useful description of structural properties in reducible coevolutionary digraphs that we apply later.
The strong component of is a maximal induced strong subdigraph of . It is maximal in the sense that if a strong and strong are connected by arcs in both directions, in which case there exists both -path and -path for every and , then they belong to the same strong subdigraph . Obviously a single-vertex is strong by definition. A digraph can be decomposed into its strong components , with each strong component being distinct by definition of it being a maximal induced subdigraph.
A reducible coevolutionary semicomplete digraph of order has a strong decomposition with a unique ordering whereby when . is the initial (terminal) strong component. In particular, the dominant solutions of are contained in its terminal strong component.
As with the case of tournaments, we first note that a reducible semicomplete digraph can be decomposed into strong components . Let the strong components be ordered as with when (Bang-Jensen et al., 1996). One can perform a contraction operation on the strong components that replaces with a single vertex and remove all parallel arcs (Bang-Jensen and Gutin, 2009). When contraction is applied to all the strong components of , the resulting digraph is called the strong component digraph. Note that is now with when ; that is, it has an acyclic ordering. The initial (terminal) strong component of is obviously the strong component whose corresponding vertex in has zero in-degree (out-degree). The same argument can be made for tournaments. However, with a reducible tournament one may have to apply contraction on transitive subtournaments, in which case the resulting digraph will be acyclic as well.
2.8 Quantitative Characterizations of Coevolutionary Digraphs
In demonstrating several structural results relating to cycle structures in coevolutionary digraphs crucial to characterizing their dichotomy in solvability, we make no assumption on availability of prior information relevant to uncovering those structures. Where more information becomes available, we can make quantitative characterizations relevant to the complexity of those cycle structures and further uncover their higher organizations. In particular, we first study how those cycles of various lengths can be packed in a manner that results in a strong decomposition of the coevolutionary tournament. We present the following result.
The first statement on score sequences of coevolutionary tournaments is a direct consequence of a theorem by Landau (1953) (Bang-Jensen and Gutin, 2009).7 The second statement on score sequences of irreducible coevolutionary tournament is a direct consequence of a theorem by Moser and Harary (1966) (Bang-Jensen and Gutin, 2009). The final statement immediately follows from the second statement, thereby providing conditions for reducibility of coevolutionary tournaments (Moon, 1963).
Lemma 29 makes the only assumption of a labeled tournament (its score sequences are known). There are several consequences to this result. A coevolutionary tournament of order is transitive if it has a score sequence of whereby for each , . Lemma 29 can be used to identify strong components in a reducible coevolutionary tournament. The idea is that to look for successive values of for which the equality holds. This follows from Lemma 25 on the decomposition of tournaments into subtournaments . A strong subtournament will be sandwiched between strong or transitive subtournaments, .
One can formalize this as follows. We consider a coevolutionary tournament with its solution set indexed according to its score sequence. Then, let , , and for with . The subtournament induced by the subset is a strong subtournament . Furthermore, the score sequence that is restricted to the strong subtournament itself is given by (Pirzada, 2012). If is irreducible, the procedure will just return a single strong component—the entire tournament itself.
Finally, we will exploit Lemma 29 to construct an index relevant to characterizing complexity of cycle structures in coevolutionary digraphs. We give a precise definition for the index as follows.
The index ranges from 1 to . Essentially, it is a count of the number of strong components in the coevolutionary tournament. When this index is used together with structural results such as Lemma 28 that provide qualitative descriptions of tournament decomposition, it provides a useful macroscale indication of cycle structure complexity in coevolutionary tournaments that is summarized as a simple quantity. if and only if the tournament is strong, while if and only if the tournament is transitive (each vertex is a strong component and all these vertices have an acyclic ordering). A value of closer to indicates the presence of many shorter length cycles that can be grouped separately into different subtournaments. When is close to one, the tournament has longer length cycles between to . can be computed quickly (the computation cost is linear with the order of the tournament with precomputed s). Later, we demonstrate how this index is used in computational studies.
3 Coevolutionary Processes on Digraphs
We introduce the second part of our framework and show that coevolutionary processes can be formulated naturally as sampling processes on digraphs. For simplicity and to allow for in-depth analysis, we consider population-one coevolutionary systems, which corresponds to random walks on digraphs. These would cover various coevolutionary systems such as those that are used in the context of learning game-playing strategies through self play. We will formulate these random walks as discrete time Markov chains operating on coevolutionary digraphs. This will allow us to study how specific structures in coevolutionary digraphs can affect the dynamics of coevolutionary processes.
3.1 Coevolutionary Processes as Random Walks on Digraphs
Algorithm 1 describes a standard random walk on labeled digraphs . It starts at a random vertex and in each subsequent step, randomly moves to an out-neighbour of the current vertex. This requires prior information on connectivity structures (out-neighbourhoods of all vertices in ). In practice, one would implement a random walk on unlabeled digraphs such as the (1+1)CEA (Algorithm 2). In the (1+1)CEA, the random move in each step is made through a two-stage process of variation (generates a random offspring from the neighbourhood of the parent vertex) and selection (moves to the offspring vertex if it dominates the parent vertex). Unlike a random walk, there is a chance the (1+1)CEA stays at the current vertex in a step because it chooses one of its in-neighbours (the probability is proportionate to the order of the in-neighbourhood). Note as well that in practice a (1+1)CEA can apply an arbitrary stopping criterion (finite number of steps as in Algorithm 2).
3.2 Formulating the Coevolutionary Markov Chain
Regardless of how random walks on digraphs are implemented algorithmically, our focus is on the underlying stochastic processes that generate those walks. We can exploit various tools for quantitative analysis of coevolutionary processes consistent with the qualitative characterizations we have presented earlier on coevolutionary problems. We first motivate formulating coevolutionary processes on digraphs as a specific class of stochastic processes—discrete time Markov chains. We will refer readers to Norris (1998) as an introductory text and Meyn and Tweedie (2009) for more advanced materials on Markov chains. We will provide relevant materials to ensure this article is self-contained.
A discrete time stochastic process is a collection of of random variables with each taking values in a countable set at time . is the set of natural numbers that include 0. A random walk is a discrete time process that takes a step on the digraph from to . Each takes values from the state space corresponding to the finite set of vertices of the digraph . Each state is the distinct vertex .
Constructing coevolution as a discrete time process is technically involved as one needs to work on probability spaces. In general, one first prescribes the structural definition for the sample path behaviors and the dynamics of steps on the digraph through time to define the probability laws that govern the evolution of the chain . Even in the usual case where sequences take discrete values in , developing a complete, functional description of the distribution of the random walk on most digraphs other than those with structures that strictly constrain its sample path behaviors (e.g., acyclic tournaments) is difficult. Instead, we use the observation that coevolutionary processes satisfy the Markov property—the step to a future state only depends on the current state.8 We exploit this to define and construct the probabilistic motion of the coevolutionary process on digraphs.
We first note the following definition and facts of a Markov chain.
We formulate and define the Markov chain for a coevolutionary process on digraphs as follows.
Let be a coevolutionary digraph having a finite vertex set . A coevolutionary Markov chain is a random walk on with initial distribution over and Markov transition matrix on the state space satisfying
, where and , and
, where for every row is a distribution whereby and .
Guarantees that such a coevolutionary Markov chain defined on the state space , initial distribution and transition matrix are given by the Existence Theorem of Markov chains on countable state space (Chung, 1960). By definition a coevolutionary digraph has a discrete with finite number of elements, which is a countable set.
3.3 Coevolutionary Markov Chain Examples
We construct four coevolutionary Markov chains (two for each case of transitive and regular tournaments). The initial distribution can be arbitrary so we just need to construct the appropriate Markov transition matrix. These examples highlight cases where we can do the constructions formally and later in Section 4.4 to perform quantitative analysis without resorting to sampling and large-scale computations as is often the case in real-world problems. The first example is the random walk on labeled transitive coevolutionary tournaments. Let be a transitive coevolutionary tournament with vertices indexed according to its score sequence . We simplify our notation and let be the Markov transition matrix for this coevolutionary Markov chain on .
4 Quantitative Characterizations of Coevolutionary Markov Chains
In this section, we first demonstrate the link between structural properties in the coevolutionary digraphs and their impact on coevolutionary processes that operate on those digraphs. Subsequently, we will introduce and develop quantitative characterizations that are relevant to those coevolutionary Markov chains.
4.1 Class Structures
One fundamental notion that can help us understand the dynamics of a coevolutionary Markov chain is the communication classes on its state space. We can analyze a large and complex coevolutionary Markov chain by breaking it into smaller pieces having shared properties (states within them are equivalent in some sense). Let be a coevolutionary digraph. is the Markov transition matrix of the coevolutionary Markov chain operating on . Let and be two distinct states .
(leads to) if and only if there is a positive probability that starting at the Markov chain reaches after -steps, for some .
The relation is transitive given the property of the -step transition probability (as indicated by Equation 4) so that one can compute for some and . By definition, , in which case the relation is reflexive.
(communicates with ) if and only if and .
In this case, the relation is symmetric and transitive. As such, the relation forms an equivalence relation on , and allows one to partition into disjoint subsets that are called communicating classes (or equivalence classes) (Chung, 1960).
Let be the class containing . A state space consisting of a single class (and the Markov chain operating on ) is said to be irreducible.
There are intimate relationships between the class structure of a coevolutionary Markov chain and the connectivity structure of the coevolutionary digraph that the chain operates on. One obvious connection is the notion of irreducibility.
A coevolutionary Markov chain is irreducible if it operates on an irreducible coevolutionary digraph.
An irreducible digraph is strongly connected whereby for every , there is an -path and -path. This implies that for all for a coevolutionary Markov chain operating on . There is only a single communicating class , in which case the coevolutionary Markov chain is irreducible.
States can be classified as essential or inessential. A state is essential if and only if for all , implies . Otherwise, is inessential when there is a such that but .
The property that a state is essential or inessential is a class property (all the states in a communicating class share the same property). An essential state cannot lead to an inessential state. Crucially, an essential class is by definition minimal closed – it is closed in the sense that and minimal in that it does not contain a proper subset that is closed (Chung, 1960). In other words, a coevolutionary Markov chain will be absorbed into an essential class upon entering it (i.e., it will be unable to leave it once it first enters it). This leads to the following lemma.
For a reducible coevolutionary Markov chain, there is only one absorbing class. The absorbing class consists of states corresponding to dominant solutions in the coevolutionary digraph.
A coevolutionary Markov chain operating on a reducible coevolutionary digraph is reducible. By Lemma 28, a reducible coevolutionary semicomplete digraph will have a terminal strong component that dominates all other remaining vertices. States corresponding to the terminal strong component form an essential communication class that is also absorbing.
Consider a reducible coevolutionary Markov chain with Markov transition matrix having an absorbing class . Let be the matrix restricted to states in . Then, there exists an irreducible coevolutionary Markov chain with state space restricted to and a Markov transition matrix given by .
This is a direct consequence of Proposition 4.1.2 (page 79 in Meyn and Tweedie, 2009).
It remains then to investigate the behavior of the coevolutionary Markov chain on and the manner in which it gets absorbed to . The next section will provide the means for which this can be done quantitatively.
4.2 Hitting Times
The analysis of the behaviors of the chain is often concerned with its distributions at certain random times as it evolves on . Additional tools are needed for which these behaviors can be described in a precise and quantitative manner. One of the quantities refers to the random times with which visits certain states in .
One can also define the first visit to to only consider positive times (Meyn and Tweedie, 2009).
Obviously, is the case of with .11 Another quantity relates to the number of visits to some states.
The occupation time counts the number of visits of a coevolutionary Markov chain to after time zero. It is crucial in the following characterization of the set .
is uniformly transient if there is a finite for all . Otherwise, is recurrent if for all .
This can be extended to individual states as well, i.e., a state is recurrent if (Meyn and Tweedie, 2009). This leads to the following lemma.
Consider an irreducible coevolutionary Markov chain on . Every state is recurrent.
From Lemma 37, an irreducible operates on that coincides with an irreducible coevolutionary digraph . Such a digraph is Hamiltonian from Lemma 26, in which case there is a closed finite-length -path for all . This implies that for all . Applying Proposition 8.1.3 in Meyn and Tweedie (2009), we have that . As such, all is recurrent.
How about the case of a reducible coevolutionary Markov chain? Obviously, its state space can be partitioned into two classes—a closed absorbing class of recurrent states and another consisting of transient states only. As in the context of analysis of Evolutionary Algorithms (EAs) (He and Yao, 2003), studying the expected (mean) hitting time of an absorbing coevolutionary Markov chain is crucial. Unlike EAs, complete connectivities in the coevolutionary digraph means although it is a rare event for to make a one-step walk to . The following gives a precise definition for this quantity (Norris, 1998).
4.3 Invariant Measures—Stationary Distribution
This section is concerned with characterizing the coevolutionary Markov chain in the context of the long-time properties of . This is relevant for irreducible coevolutionary Markov chains as well as reducible chains (after the chain enters the absorbing class). We first introduce the notion of an invariant measure (Norris, 1998).
A that is defined on and satisfies the invariant property would represent the long-term limiting distribution of the coevolutionary Markov chain. In addition, such a distribution can be described in the following equivalent terms:
It is known that for an irreducible Markov chain on countable state space , its stationary distribution can be calculated from the expected return times (Levin et al., 2009). The general intuition is that if one breaks up the trajectory of into identically distributed segments through visits to and from some arbitrary states, the average fraction of time that is spent on a state by each segment would be similar to the long-term fraction of time spent on by . Furthermore, starting from any arbitrary distribution , the chain would converge to its stationary distribution in the long run (Norris, 1998; Levin et al., 2009). For coevolutionary Markov chains, we have the following results.
Consider an irreducible coevolutionary Markov chain with Markov transition matrix . Then, there is a unique invariant probability distribution on where and such that . Furthermore, .
This is a direct consequence of Theorems 1.7.7 in Norris (1998) whereby an irreducible and positive recurrent Markov chain on a finite state space has a unique invariant probability measure on with . A state is positive recurrent if its expected return time is finite. By Lemma 37, obviously all s are positive recurrent since is finite.
Consider a reducible coevolutionary Markov chain with Markov transition matrix . For all inessential states , . Then, there is a unique invariant probability distribution concentrated on the absorbing class of essential states in .
Let be the set of all inessential states . The first statement on for all inessential states is a direct consequence of Proposition 1.25 (page 16 in Levin et al., 2009). This implies that the probability distribution is concentrated on the remaining essential states of the absorbing class . From Lemma 40, we can study the irreducible coevolutionary Markov chain with separately and obtain as a result of Lemma 49. Since and , then whereby all elements in are zero.
This is a direct consequence of the Convergence to Equilibrium Theorem (given as Theorem 1.8.3 in Norris (1998). We just need to show that an irreducible coevolutionary Markov chain is aperiodic. By Lemma 26, irreducible coevolutionary digraphs are vertex pancyclic, in which case the greatest common divisor to the cycle lengths is one. As such, is aperiodic.
4.4 Case Studies on Quantitative Measures of Coevolutionary Markov Chains
We first provide analytic expression for the expected hitting time of specific coevolutionary Markov chains. This case study considers both the random walk and (1+1)CEA on transitive coevolutionary tournaments. We note that the Markov transition matrices given by Equations 5 and 6 are in the form of an upper triangle matrix. It requires re-labeling the indices in so that it becomes a lower triangle matrix. For these two cases, s are also in the form of a lower triangle matrix. Crucially, each of the matrices has a so its inverse exists (Meyer, 2000).
In this second part of our case study, we calculate the stationary distribution of a specific class of coevolutionary Markov chain—those operating on regular tournaments. We can make use of structural results from digraph theory to obtain a general result on the stationary distribution of regular coevolutionary Markov chains.
Consider a coevolutionary Markov chain operating on a regular coevolutionary tournament of odd order , . This regular coevolutionary Markov chain has a unique invariant probability measure on whereby for all .
Every regular coevolutionary tournament is arc-pancyclic, which is a direct consequence of Corollary 6.6.4 in Bang-Jensen and Gutin (2009). An arc-pancyclic coevolutionary tournament is arc--cyclic for all ; that is, each arc of is contained in a cycle for each length of . Since for all in a regular , this means that there is an equal number of arcs of various lengths that start and end at each . Any random walk on would thus spend equal fraction of time for each , in which case for a tournament with vertices.
One immediate consequence of Lemma 52 is that any coevolutionary process (random walk or (1+1)CEA for example) can be defined on a regular tournament of order and that since the cycles that start from and return to each vertex are the same. We can also apply this result with Lemma 50 to calculate the stationary distribution of a reducible coevolutionary Markov chain having an absorbing class that is essentially a regular subtournament.
4.4.1 Absorbing Coevolutionary Markov Chains
Earlier result for (1+1)CEA with uniform expected hitting time for any starting vertex indicate unique properties (the diagonal elements in the probability transition matrix), which we can generalize further with the following result.
Consider a (1+1)CEA operating on a reducible coevolutionary tournament . Let consists of two disjoint subsets and , whereby . Let . Let the coevolutionary Markov chain be defined with Markov transition matrix in its canonical form . Let be the transient states and that . Let the column vector where is the expected hitting time starting from a transient state to the absorbing class . Then, .
For every transient state , note that the corresponding vertex in the co-evolutionary tournament has the same degree . Let and . Given our definition of (1+1)CEA (as in Algorithm 2), for each : (1) and (2) . The one-step transition probability from to the single absorbing state is given by . Since each one-step transition from to its outneighbour occurs with equal probability, then and . Note that . As such, each row of sums to one, i.e., .
Next, we need to show that the corresponding row sums of follow a geometric series. We need to prove that . This will give the expected hitting times calculated from the geometric series . We do this by induction. Let the column vector , with representing the th element of . The induction hypothesis is that .
Basis (): By definition . Then, .
Inductive step: We need to show that . We first note that . Let and . Let . . Since , then .
From our induction hypothesis, it can be seen that . Each row sum of then follows the geometric series . Thus, .
The procedure to show that the row sums of follow a geometric series is the same as in Theorem 53. It can be shown that . Each element of then follows the geometric series , in which case, .
4.5 Generating Random Coevolutionary Tournaments
We conclude this section with a computational study on generating random coevolutionary tournaments in a controlled manner. Potential applications include using random tournaments as benchmarks to study methodologies estimating quantities of interest in more complex and realistic coevolutionary Markov chains. We first note that the simple approach of randomly orienting edges of the underlying graphs would usually generate irreducible random tournaments (Moon, 1968). There is more than chance to generate an irreducible random tournament of order five, which increases to more than for random tournaments of order 16! Beyond that, generated random tournaments would almost certainly be irreducible. Another approach is to first generate all valid score sequences (Hemasinha, 2003), which are used to reconstruct the corresponding tournaments (Moon, 1968).
This setting can be motivated by statistical physics. Each node of the existing tournament has an ``energy'' related to its score (in-degree). The new incoming node will be more likely to orient edges towards high energy node in the existing tournament. Parameter plays the role of inverse temperature. For high temperature regimes, the energy of nodes begins to equalize and the probability of edge orientation is the same for any node in the existing tournament. In contrast, by cooling the system (increasing ), orientation of edges from the new node towards high energy nodes of the existing tournament becomes more persistent.
We have performed initial experiments to determine appropriate choices for the parameters involved. We initialize (seed) the graph growth with transitive tournaments. For the statistics , we use the corresponding node ranks with , . Tournaments of order are generated from initial ones of order . Each experiment set is repeated 100 times, thus generating 100 tournaments randomly and independently. The results are summarized in Figure 3. Obviously, the number of strong components are higher when the initial tournament seeds are larger as shown in Figure 3b.
We next introduce into our 1000-node tournaments an additional single dominant (1001-th) node that has all the edges from the original 1000 vertices oriented towards it. This construction makes the tournaments reducible and so we can compute the expected hitting time from the weakest node (with the lowest score) to the dominant (absorbing) node . We now study the impact of increasing transitive structures, measured by the number of strong components (reflected in the complexity (Landau's) index (Equation 1)), on dynamics of the coevolutionary process, measured by the expected hitting times of the dominant node.
First, we show results for the experiment that uses the standard random walk (Algorithm 1) with initial seed tournament of order 10. We can clearly see signatures of higher and lower temperature regimes in Figure 4a. In the high temperature regime (), all the base 1000-node tournaments are irreducible, hence there is only one strong component. Together with the dominant 1001-th node, there are two strong components. There is little variability of hitting times (around 492–497), which reflects limited variability of the internal structure of the irreducible 1000-node base tournaments. For comparison, the expected hitting time is exactly 500 for a regular 999-node base tournament. In the low temperature regime (), prominent transitive structure dominates the base tournaments and the hitting times range in . For comparison, the expected hitting times would be slightly less than 10 for a vertex pancyclic 1000-node base tournament (which has the longest transitive structure) and around 7.4855 for a transitive 1000-node base tournament.
As in most complex systems, in between these two temperature regimes sits an intermediate temperature regime that exhibits the most interesting behaviour. Full spectrum of possible values of Landau's index and expected hitting times can be observed. Indeed, this is the case where transitive structures co-exist with a wide range of hitting times, reflecting different degrees of cycle structures in the strong components. Figure 4 illustrates the relationship between the (inverse) temperature and the hitting times of the dominant 1001-th node. As expected, while extremely high temperatures lead to irreducible base tournaments with intricate cycle structures and long hitting times, very low temperature regimes lead to base tournaments with prominent trivial transitive structures, yielding minimal hitting times. It is also natural to expect that, on average, the expected hitting times will decrease with decreasing temperature, since lower temperatures induce more transitive structures in the base tournaments. The most “complex” and interesting regimes lie in between these two extremes at .
Theorem 53 will indicate that the expected hitting times for (1+1)CEA operating on these generated tournaments to be the same as their orders (1001 in this case), which are verified by repeating the experiment with (1+1)CEA. Computations for the expected hitting times are dependent on the probability transition matrix associated with the coevolutionary Markov chain under study. Obviously, different constructions of coevolutionary processes—standard random walk against (1+1)CEA—operating on the same tournaments can lead to different outcomes in the expected hitting time analysis. In contrast, a digraph-theoretic-based approach such as the Landau's index is unaffected by the specific constructions of coevolutionary processes. There are potential ramifications on the construction of sampling-based indices (e.g., hitting times) for analysis of structures in coevolutionary tournaments. Our controlled computational study suggests that more information related to the structures of the tournaments under study may need to be incorporated into those indices so that they are able to uncover those relevant structures. For example, a standard random walk incorporates information of connectivity structures (i.e., out-neighbourhoods of vertices) directly in contrast to the (1+1)CEA.
5 Discussion and Conclusion
In this study, we have addressed the issue of studying rich cycle structures associated with coevolutionary systems that use pairwise-comparison-based fitness measures. We have developed a framework that fully captures the underlying structures in coevolutionary problems, and we represent them as cycle structures in digraphs so that we can begin to uncover their effects on coevolution. The digraph representation allows us to give precise characterizations of cycles of different lengths (from 3-cycles to Hamilton cycles that go through every vertex in the digraph) that can occur in coevolutionary problems.
Although these cycle structures appear complex at first given the variety of cycles of different lengths and the manner they can be oriented in digraphs, our study has revealed that coevolutionary digraphs admit a simple qualitative characterization—the corresponding coevolutionary problem is either solvable or not. This has a significant implication on coevolution when it is used as a search procedure. For nonsolvable problems, the presence of Hamilton cycles in the associated digraphs means that coevolution will cycle indefinitely. For each solvable coevolutionary problem, there is a subset of solutions that dominate all other solutions in the solution space. In the context of games, coevolution would be able to search for these dominant strategies in the long run.
The second part of our framework has formalized coevolutionary processes as random walks on digraphs. These random walks are formulated as discrete time processes having the Markov property, which have allowed us to exploit the qualitative characterizations we have made on coevolutionary digraphs to construct relevant quantitative tools for analysis of those processes. We have shown that coevolution of solvable problems corresponds to an absorbing Markov chain with a single absorbing class. For this case, we have shown that the expected hitting time to the absorbing class is the quantity of interest. The other quantity is the limiting invariant distribution of the Markov chain. Although this is of principal interest for coevolution of non-solvable problems, the quantity can also be computed for the case of solvable problems as a result of Lemma 50.
We have demonstrated our quantitative tools through a combination of formal and computational case studies. We have derived expected hitting times and stationary distributions of specific coevolutionary Markov chains operating on transitive and regular tournaments. We have suggested an index for characterizing complexity in coevolutionary problems, and we show how a hierarchy of increasingly complex coevolutionary problems can be generated in a controlled manner that is motivated by statistical physics.
We believe this framework would provide the foundation for existing tools and the means to develop new tools for analysis and design of coevolutionary systems that use pairwise-comparison-based fitness measures. As a case in point, coevolution could appear to be trapped in cycles even when it is applied on solvable problems (as observed in various studies such as Chong et al., 2012). Although the associated reducible coevolutionary digraphs can be decomposed into acyclically ordered strong components (by Lemma 28), some of these strong components can be very large digraphs. In the extreme case, a reducible coevolutionary digraph of order has a strong component of vertices. One possibility is that coevolution would cycle for a long time before jumping out and find the single dominant vertex. The other possibility is that the -sized component is the dominant one, in which case coevolution would seem to cycle indefinitely even though it has found (parts of) the dominant component. In the more realistic case, one could reframe the pairwise-comparison-based fitness measure in the context of pareto dominance and make the appropriate change to coevolution to search in the direction of more dominant strong components (Pareto fronts with higher true fitness values in the solution space) (de Jong and Pollack, 2004; de Jong, 2007).
The introduction of this framework for coevolutionary systems provides several directions for future studies. One of them is to provide a more definitive characterization of problem hardness affecting coevolutionary systems that include studies on local structures (internal cycle structures within strong components) of coevolutionary digraphs.
Beyond the restricted class of coevolutionary systems we have studied here, the mathematical methodologies we have introduced could be adopted in other formulations of coevolutionary systems. For example, those with population size greater than one and for which coevolutionary fitness measures other than pairwise-comparison (such as the use of average game payoffs) can be investigated. In this case, the entity of interest is the individual state (vertex) representing a finite collection (population) of pure strategies in the context of game-play. The advantage of such an approach is a more informative state space description. In contrast, in evolutionary game theoretic setting studied in the framework of continuous dynamical systems, one usually assumes infinite populations. However, in cases of complex, chaotic coevolutionary dynamics, this can be problematic (Ficici et al., 2005; Tiňo et al., 2013).
Our study is by no means the first one to consider representation of the structure of pairwise relations within the problem to be solved through a digraph. For example, in the multicomponent problems (Bonyadi et al., 2013; Klamroth et al., 2017), the digraph represents pairwise coupling between individual optimization problems that define together the global compound optimization problem to be solved. Typically, one can expect a small number of subproblems (vertices) and there may not exist any coupling between a particular pair of subproblems. In our case, the vertices represent strategies from a potentially huge set of strategies that can be chosen. In addition, the strategy set is fully coupled; that is, between any two different strategies there exists an arc. Most importantly, direct application of Markov chains on such a digraph has the interpretation of a random walk on the strategy space following the outcomes of individual games.
This work was supported by a H2020-MSCA-IF-2014 grant (No. 657027) on “COEVOLFRAMEWORK—Unified Framework for the Analysis of Co-evolutionary Systems.”
We do consider loops later when we introduce Markov chains, which essentially are weighted digraphs with specific constraints.
Cycles of lengths two and one can be defined as well and usually in context of digraphs that admit specific connectivity structures. A 2-cycle is the case whereby two adjacent vertices is completely bioriented via and . A 1-cycle is simply the case of a self-loop .
Implicit in our notation are the corresponding sets of arcs in and satisfying constraints for tournaments and semicomplete digraphs, respectively.
Standard digraph terminology defines domination in the reverse order, that is, dominates is denoted as . Our choice to define domination in this manner becomes clear in the next section when we discuss coevolutionary processes on digraphs and their Markov chain models.
We are not describing how one can find the best strategy but how relations of strategies can be captured, for example, transitive strategic games represented by acyclic orientations of complete graphs.
A coevolutionary tournament with one vertex is irreducible since it is a strong tournament by definition. A tournament with two vertices is reducible since there is always one vertex that dominates the remaining vertex in the tournament.
Landau's theorem can be generalized further. For example, Reid and Zhang (1998) provided necessary and sufficient conditions for a sequence of integers to be the score sequence of semicomplete digraphs. Moon (1963) showed that the original theorem can be extended to the case whereby one can assign a real number to the outcome of the pairwise comparison (game) between a pair of vertices (strategies). That is, for a game between two strategies ( and ), is the score for , and is the score for .
Loosely speaking, the dynamics of the coevolutionary process as it walks on the digraph is such that it restarts in every step. For example, with the (1+1)CEA, the probability that it steps from a particular vertex to an out-neighbour is independent of the time (generational) step ; that is, the probability is the same whether it is at the start or , which is generations later.
First hitting and return times are stopping times in the sense that the event depends only on for , which follows from the strong Markov property (e.g., Theorem 1.4.2 in Norris (1998)). The Markov property is then said to hold for those random times. Furthermore, given that the coevolutionary Markov chain is discrete time, the strong Markov property always holds (by Proposition 3.4.6 in Meyn and Tweedie, 2009).
Given the complete connectivities of coevolutionary digraphs.