## Abstract

Studying coevolutionary systems in the context of simplified models (i.e., games with pairwise interactions between coevolving solutions modeled as self plays) remains an open challenge since the rich underlying structures associated with pairwise-comparison-based fitness measures are often not taken fully into account. Although cyclic dynamics have been demonstrated in several contexts (such as intransitivity in coevolutionary problems), there is no complete characterization of cycle structures and their effects on coevolutionary search. We develop a new framework to address this issue. At the core of our approach is the directed graph (digraph) representation of coevolutionary problems that fully captures structures in the relations between candidate solutions. Coevolutionary processes are modeled as a specific type of Markov chains—random walks on digraphs. Using this framework, we show that coevolutionary problems admit a qualitative characterization: a coevolutionary problem is either solvable (there is a subset of solutions that dominates the remaining candidate solutions) or not. This has an implication on coevolutionary search. We further develop our framework that provides the means to construct quantitative tools for analysis of coevolutionary processes and demonstrate their applications through case studies. We show that coevolution of solvable problems corresponds to an absorbing Markov chain for which we can compute the expected hitting time of the absorbing class. Otherwise, coevolution will cycle indefinitely and the quantity of interest will be the limiting invariant distribution of the Markov chain. We also provide an index for characterizing complexity in coevolutionary problems and show how they can be generated in a controlled manner.

## 1 Introduction

In this study we focus on coevolutionary systems that are specified by the: (1) *coevolutionary problem*—the set of all alternatives (e.g., strategies in a two-player game) and a pairwise preference relation indicating the superior alternative (e.g., whether when playing the game one strategy defeats another one), and (2) *coevolutionary process*—the population of alternatives undergoing a process of selection and variation that is guided only by their interaction outcomes. Coevolutionary systems are often used in competitive settings where the lack of absolute fitness measures necessitates that one exploit the strategic nature of interactions in coevolving populations. In this context, coevolutionary systems provide a natural alternative for solving test-based problems (Hillis, 1990; de Jong and Pollack, 2004) that include two-player strategic games whereby opponent strategies provide test cases for fitness evaluations (Runarsson and Lucas, 2005). In turn, game-theoretic-based frameworks have been developed to provide tools for analysis of coevolutionary systems (Ficici et al., 2005; Tiňo et al., 2013).

The use of relative and dynamic fitness evaluations in coevolutionary systems introduces unique challenges that make these systems difficult to analyze and understand in comparison to evolutionary systems. They are collectively referred to as coevolutionary pathologies. In the classic example of two-population predator-prey coevolution, the *Red Queen effect* is coined to describe the interdependence in the fitness evaluations of members in one population to the other even though there is no exchange of genetic materials (Cliff and Miller, 1995; Janssen et al., 2016). *Disengagement* occurs when there are large gaps in competence levels between the populations, leading to a loss of selection pressure, decoupling of the coevolutionary system, and then onset of random drift in the populations (Cartlidge and Bullock, 2004). For problems with intransitivity in the relationships between the solutions, *cyclic dynamics* have been observed whereby the population overspecializes to copies of a solution that is exploited by another solution later in the coevolutionary process (Darwen and Yao, 1997). In the more extreme case, coevolution *forgets* a solution it has discovered earlier—the solution that is driven to extinction is adaptively found later (Rosin and Belew, 1997).

Although there are broader implications in design of effective coevolutionary systems, pathologies are usually associated with the selection part of the coevolutionary process, particularly on the use of pairwise-comparison-based fitness measures. Most studies would frame the pathologies in specific settings for which relevant methodologies can be used for analysis. These include measurements for monitoring the progress of coevolutionary systems using generational samples of test cases (Cliff and Miller, 1995; Stanley and Miikkulainen, 2002; Bader-Natal and Pollack, 2005) and those where one has a global view and can objectively measure performance of solutions (Floreano and Nolfi, 1997; Chong et al., 2012). In Ficici et al. (2005) and Tiňo et al. (2013), a formal approach is taken to study coevolutionary dynamics. Under the usual evolutionary game theoretic setting such as the use of an infinite population, a full characterization is given on coevolutionary dynamics (as changes to the ratio of individuals adopting a pure strategy) for various dynamical maps modeling different selection operations.

Here, we focus on coevolutionary systems that use pairwise-comparison-based fitness measures. However, it remains an open challenge to study this restricted class of coevolutionary system since the rich underlying structures associated with pairwise-comparison-based fitness measures are often not taken fully into account. Although cyclic dynamics have been shown in coevolutionary systems, most past studies such as Samothrakis et al. (2013) would focus on short cycles (of length three). Longer length cycles can occur in coevolution. Crucially, there is no complete characterization of cycle structures and their effects on coevolution. Our motivation is to develop a new framework to address this issue formally. Our approach uses the digraph representation for coevolutionary problems to capture the full structures in the binary relations between candidate solutions. Coevolutionary processes are modeled naturally as discrete time Markov chains operating on discrete state spaces (the sets of vertices of digraphs). In this manner, different coevolutionary algorithms would correspond to different implementations of random walks on digraphs.

We first show that coevolutionary problems admit a qualitative characterization: a coevolutionary problem is either solvable (the digraph is reducible with a subset of solutions that dominates the remaining solutions of the solution space) or not. This has an implication on coevolutionary search where we further develop the framework to construct quantitative tools for appropriate analysis and demonstrate their applications through case studies. We will show that coevolution of solvable problems corresponds to an absorbing Markov chain for which we can compute the expected hitting time of the absorbing class. Otherwise, coevolution will cycle indefinitely and the quantity of interest will be the limiting invariant distribution of the Markov chain. We also suggest an index for characterizing complexity in coevolutionary problems and show how a hierarchy of increasingly complex coevolutionary problems can be generated in a controlled manner.

This article is organized as follows. In Section 2, we formulate the digraph representation and provide several results characterizing cycle structures in coevolutionary problems. Section 3 formulates coevolutionary processes as random walks on digraphs and provides examples how specific Markov chains are constructed from coevolutionary algorithms operating on digraphs. Section 4 introduces relevant quantitative characterizations for coevolutionary systems (expected hitting times and stationary distributions of Markov chains). These quantitative tools will be demonstrated through theoretical and computational case studies. Section 5 will discuss the main findings and wider implications of our framework with remarks for future studies.

## 2 Digraph Representation of Coevolutionary Problems

We first present basic definitions and facts on digraphs relevant to formulating our framework to make this article self-contained. Our notation and terminology follow standard digraph theory texts and readers can refer to Bang-Jensen and Gutin (2009) for detailed expositions of the following. Our focus is to show how digraphs provide appropriate mathematical structures that fully capture the pairwise preference relations between objects of interest that one would encounter in typical coevolutionary problems such as games. Later, we would exploit classic results in digraph theory to provide a complete characterization of cycle structures in coevolutionary problems.

### 2.1 Digraphs—Definition, Notation, and Terminology

A *digraph*$D=(V,A)$ consists of two sets: (i) a non-empty, finite set $V(D)$ of elements called *vertices*, and (ii) a finite set $A(D)$ of elements corresponding to ordered pairs $(u,v)$ of *distinct* vertices, $u,v\u2208V(D)$, called *arcs* (or *directed edges*). The arc set is a *binary relation* on the vertex set ($A$ is a subset of the *Cartesian product* of $V$ with itself, $A\u2282V\xd7V$).

The *order* of a digraph $D=(V,A)$ is given by the *number of vertices*$|V|$ (often denoted as $|D|$). The *size* of $D$ is given by the *number of arcs*$|A|$.

A graph $G=(V,E)$ consists of: (i) a non-empty, finite vertex set $V(G)$, and (ii) a finite set $E(G)$ corresponding to unordered pairs ${x,y}$ of *distinct* vertices, $x,y\u2208V(G)$, called *edges*. The *underlying graph*$UG(D)$ of a digraph $D=(V,A)$ is the graph with the same vertex set $V$ and the corresponding edge set $E$ whereby arcs are replaced by edges with no parallel edges and loops.

We often use a simpler notation and denote the arc $(u,v)$ as $uv$. The tail $u$ and head $v$ are the *end-vertices* of $uv$. *Adjacency* between end-vertices $u$ and $v$ implies an arc $uv$ or $vu$. As in Bang-Jensen and Gutin (2009), we consider the digraph $D=(V,A)$ without *parallel* arcs between adjacent end-vertices (arcs having the same tail and same head) and *loops* (the arc $uu$ where the tail and head is the same vertex).^{1}

The definition of digraphs and in particular the binary relation on the vertex set that can be constructed allows for rich structures to be captured. We will consider notions of orientation, biorientation, complete, and semicomplete.

The *orientation* of a graph is the digraph where each edge corresponding to a pair of adjacent vertices $u,v\u2208V$ is connected by an arc (either $uv$ or $vu$ only). The *biorientation* of a graph is the digraph where each edge in the graph is replaced by the arc $uv$ or $vu$ or both arcs $uv$ and $vu$.

Obviously, any digraph is a biorientation. An oriented graph is a specific biorientation since any pair of adjacent vertices is connected by an arc only. In a *complete* graph, every pair of distinct vertices is adjacent.

A *semicomplete digraph* is a biorientation of a complete graph (since not all pairs of distinct vertices are completely bioriented, in which case they are only oriented). A *tournament* is a complete oriented graph.

Tournaments form a special and well-studied class of semicomplete digraphs (Bang-Jensen and Gutin, 2009). Tournaments and their generalizations (semicomplete digraphs) would be the focus later in this article.

### 2.2 Neighbourhoods and Degrees

*neighbourhood*of a vertex $u\u2208V$, $ND(u)$, refers to the subset of $V$ whereby its elements are adjacent to $u$. It consists of the union of the

*out-neighbourhood*$ND+(u)$ and

*in-neighbourhood*$ND-(u)$ of $u$:

We note $V-{u}\u2282V$ results from the set *difference* between $V$ and $u$ in the usual manner. The vertices in $ND+(u)$, $ND-(u)$, and $ND(u)=ND+(u)\u222aND-(u)$ are called *out-neighbours*, *in-neighbours*, and *neighbours*, respectively (Bang-Jensen and Gutin, 2009).

A related notion of neighbourhoods is the *degrees* in a digraph.

*out-degree*$dD+(u)$ (

*in-degree*$dD-(u)$) of a vertex $u$ is the number of outgoing (incoming) arcs in the digraph $D$ where $u$ is the tail (head) for each of these arcs. They are given by $dD+(u)=|(u,V-{u})D|$ and $dD-(u)=|(V-{u},u)D|$. The degree of $u$ is given by $dD(u)=dD+(u)+dD-(u)$.

### 2.3 Walks, Paths, and Cycles

Consider a digraph $D=(V,A)$, whereby vertices $vi$ and arcs $ai$ from $D$ are labeled in a manner such that the tail and head of $ai$ is $vi$ and $vi+1$, respectively, for every $i=1,2,3,\u2026,k-1$ (Bang-Jensen and Gutin, 2009).

A $(v1,vk)$-*walk*$W$ in the digraph $D$ from $v1$ to $vk$ is the alternating sequence of vertices and arcs $v1a1v2a2v3\u2026ak-1vk$. The set of vertices of the walk is $V(W)={v1,\u2026,vk}$ while the set of arcs of the walk is $A(W)={a1,\u2026,ak-1}$. $W$ is said to be *closed* if $v1=vk$, otherwise it is *open*. The length of a walk is the number of arcs in it and by definition of $W$ is simply $k$.

A $(v1,vk)$-*path*$P$ is a walk where all the vertices in it are distinct. A *k-cycle*$C$ is a closed $(v1,vk)$-walk of length $k\u22653$ on the $(v1,vk-1)$-path.^{2}

We usually denote $W$, $P$, and $C$ without the arcs, $v1v2v3\u2026vk$, since $ai$ is defined in the context of $vivi+1$ (Bang-Jensen and Gutin, 2009). $P$ and $C$ on $D$ are by definition *subdigraphs* of $D$, e.g., $V(P)\u2286V(D)$, $A(P)\u2286A(D)$, and $A(P)={uv:u,v\u2208V(P)}$.

The path $P$ is said to be *Hamilton* if $V(P)$ is the same as $V(D)$. A Hamilton cycle $C$ has $V(C)=V(D)$. A digraph $D$ is said to be *Hamiltonian* if it contains a Hamilton cycle.

### 2.4 Classes of Coevolutionary Problems as Digraphs

We state how coevolutionary problems can be represented as digraphs in the following.

A coevolutionary problem is defined as an ordered pair $C=(S,R)$ where $S$ represents the solution set and $R$ is the pairwise preference relation on $S$. This is represented as the digraph $DC=(VS,AR)$. The vertex set is the solution set $VS=S$ (each distinct vertex corresponds to a single unique solution). The arc set is the binary relation on $S$, $AR=R$ (each biorientation corresponds to the pairwise preference relationship between each pair of solutions).

The underlying graph $UG(DC)$ is complete since there is a relationship between every pair of solutions in a coevolutionary problem. Each problem is specified by the number of unique solutions and associated preference relation on those solutions, which is described by the biorientations on $UG(DC)$. We use the term *coevolutionary digraph* to emphasize the digraph representation of a coevolutionary problem. It should be clear from the context of subsequent presentations whether we are referring to coevolutionary problems or just the digraphs as mathematical structures.

For this study, we consider two general classes of coevolutionary problems with their corresponding digraph representations.

Consider the coevolutionary problem where $VS(n)$ is the solution set with $|VS(n)|=n$. Its underlying graph $UG(DC)$ is complete. Then: (i) a *coevolutionary tournament* is an orientation of $UG(DC)$, and (ii) a *coevolutionary semicomplete digraph* is a biorientation of $UG(DC)$. For each $n=3,4,5,\u2026$, let $T(VS(n))$ and $SD(VS(n))$ be the sets of all tournaments and semicomplete digraphs of order $n$, respectively. $T(VS(n))$ and $SD(VS(n))$ represent the family of all coevolutionary tournaments and semicomplete digraphs with $n$ candidate solutions, respectively.^{3}$SD(VS(n))$ generalizes $T(VS(n))$ since adjacent vertices in $VS(n)$ can be completely bioriented. That is, $T(VS)\u2282SD(VS)$.

To provide a direct context, there is a wide range of coevolutionary problems that can be modeled as *two-player strategic games*. In each case, the game consists of a set of *pure strategies* and a *dominance relation* on those strategies. Both players have access to the same *strategy set*. Each player will select a single pure strategy from the strategy set and then implement that strategy during game-play. The game outcome is decided by the dominance relation on the two pure strategies implemented for the game-play. Some games have only *win-lose* game outcomes—the winning strategy is said to dominate the losing strategy. Other games (two-player board games such as chess) have a *win, lose, or draw* outcome. Digraphs naturally represent these coevolutionary problems. The set of vertices of the digraph is the strategy set. The dominance relation is captured and described by the set of arcs. We do not consider game-plays between copies of the same pure strategy for simplicity—there is no self-loop in the digraph.

For any pair of strategies $u,v\u2208VS$, the *winning strategy*$v$*dominates* a *losing strategy*$u$. This relation is represented by the orientation $v\u2190u$ (or $u\u2192v$ if we say $u$ is *dominated* by $v$).^{4} A draw is represented by the complete biorientation $u\u21c4v$.

The notion of domination can be extended to subsets of $VS$ (a vertex is a subset of $VS$ with a single element). Consider two disjoint subsets of $VS$, $VS1$ and $VS2$, whereby $VS1\u2229VS2=\u2205$. $VS2\u21d0VS1$ denotes that all vertices $v\u2208VS2$ dominate all vertices $u\u2208VS1$. We say $VS2$ is the *dominant* subset with respect to $VS1$. Conversely, $VS1\u21d2VS2\u2261\u2200v\u2208VS1\u2200u\u2208VS2(v\u2192u)$. The tournament provides a representation for *all* strategic games with win-lose game outcomes. Semicomplete digraph as a further generalization of the tournament can be used to represent any strategic game with win, lose, or draw outcomes.^{5}

### 2.5 Structures and Higher Organizations in Coevolutionary Digraphs

In practice, one often is confronted with a very large coevolutionary digraph with little (connectivity) information. A coevolutionary tournament with $|V|$ candidate solutions has $2(|V|)(|V|-1)/2$ pairs of comparisons. This exponential increase in pairwise comparisons is further compounded by a high (and possibly exponential) growth rate in the number of *parametric* representations of candidate solutions. Consider a simple *normal form* game such as the *iterated prisoner's dilemma* restricted to just memory-one, reactive (self and opponent's past plays) strategies. If one uses a direct look-up table representation, a $q$-choice game would generate $qq2+1$ different combinations of strategy tables (Chong et al., 2012). A two-choice game would generate eight strategy tables and 784 pairwise comparisons, which increase to at least $6.18\xd71028$ strategy tables and $3.66\xd710114$ pairwise comparisons for a six-choice game in a full tournament.

Performing a brute-force computation on a very large coevolutionary digraph (e.g., decide if the problem has a single dominant solution) quickly becomes a very expensive task. Instead, we study specific structures leading to some form of higher organizations in coevolutionary digraphs that can provide useful characterizations of coevolutionary problems. These structures emerge from constraints that could be placed on the biorientations of coevolutionary digraphs.

### 2.6 Connectivity Structures in Digraphs

We first define basic connectivity structures in digraphs before introducing qualitative characterizations of tournaments having specific connectivity conditions.

A digraph $D$ is said to be *strongly connected* (or *strong*) if for each pair of distinct vertices $u,v\u2208V(D)$ there exist both $(u,v)$-path and $(v,u)$-path.

Given a pair of distinct vertices $u,v\u2208V(D)$, $v$ is said to be *reachable* from $u$ if and only if there is an $(u,v)$-path in $D$. A vertex is reachable from itself and so by definition a digraph with a single vertex is strong. A strong digraph will have each vertex reachable from all other vertices. There exists a closed Hamilton walk that cover all vertices in the digraph. The following describes strong digraphs having specific cycle structures.

A digraph $D=(V,A)$ of order $n$ that has $k$-cycles of all lengths $k=3,4,5,\u2026,n$ is said to be *pancyclic*. The digraph is further said to be *vertex-pancyclic* if for every $v\u2208V(D)$ and every $k\u2208{3,4,5,\u2026,n}$, there is a $k$-cycle containing $v$.

*Acyclic* digraphs are those without cycles and are defined formally as follows.

Let $v1,v2,v3,\u2026,vn$ be an ordering of $V(D)$. If $v1,v2,v3,\u2026,vn$ is such that for all arcs $(vi,vj)\u2208A(D)$, $i<j$, the digraph $D=(V,A)$ is said to have an *acyclic ordering*. Every acyclic digraph has a vertex with zero in-degree $dD-(v1)=0$ and a vertex of zero out-degree $dD+(vn)=0$.

Every acyclic digraph has an acyclic ordering (Bang-Jensen and Gutin, 2009).

A digraph $D=(V,A)$ is *transitive* if for every pair of distinct arcs $(u,v),(v,w)\u2208A(D)$, $u\u2260w$ and $(u,w)\u2208A(D)$.

A tournament is transitive if and only if it is acyclic (Bang-Jensen and Gutin, 2009).

For each $n=3,4,5,\u2026$, let $T(VS(n))$ be the set of all tournaments of order $n$. Vertices for each tournament $T(VS(n),A)\u2208T(VS(n))$ can be labeled (indexed) in a manner so that $v1,v2,v3,\u2026,vn$ is ordered according to $dT-(v1)\u2264dT-(v2)\u2264dT-(v3)\u2264\cdots \u2264dT-(vn)$. This is done using the score sequence.

The *score sequence* of a tournament $T(VS(n),A)\u2208T(VS(n))$ is the $n$-tuple $(s1,s2,s3,\u2026,sn)$ with the constraints $s1\u2264s2\u2264s3\u2264\cdots \u2264sn$. Each $si$, $i=1,2,3,\u2026,n$, corresponds to the number of vertices that $vi$ dominates, i.e., $si=dT-(vi)$.

The following two definitions (Moon, 1968) are crucial in the next subsection where we present our results on how vertices in a digraph can be grouped or classified in some manner that reveals useful structures.

Consider a *vertex partition* of vertices $VS(n)$ of a tournament $T(VS(n),A)$ into two disjoint, nonempty subsets $VS(n)1$ and $VS(n)2$ whereby $VS(n)1\u2229VS(n)2=\u2205$ and $VS(n)1\u222aVS(n)2=VS(n)$. $T(VS(n),A)$ is *reducible* if it can be vertex-partitioned such that $VS(n)1\u21d2VS(n)2$ ($VS(n)2$ is the dominant subset). Otherwise, it is *irreducible*.

Some reducible digraphs further admit vertex partition (decomposition) into multiple, disjoint subdigraphs, which is defined as follows.

A digraph $D$ can be *decomposed* into subdigraphs $D(1),D(2),D(3),\u2026,D(l)$ where $VD(1)\u222aVD(2)\u222aVD(3)\u222a\cdots \u222aVD(l)=V(D)$ and $VD(i)\u2229VD(j)=\u2205$ when $i\u2260j$.

### 2.7 Qualitative Characterizations of Coevolutionary Digraphs

We first summarize several classic results of tournament theory on connectivity structures in tournaments. The results will be presented in the original form of the (referenced) primary texts (slightly restated for consistency with our notations).

A tournament $T(VS(n),A)$ is strong if and only if it is irreducible.

Every strong tournament is Hamiltonian.

Let $T(VS(n),A)$ be a strong tournament on $n\u22653$ vertices. For every $v\u2208VS(n)(T)$ and every integer $k\u2208{3,4,5,\cdots ,n}$, there exists a $k$-cycle through $v$ in $T(VS(n),A)$. In particular, $T(VS(n),A)$ is Hamiltonian if and only if it is strong.

We exploit these theoretical results and provide useful qualitative characterizations to the family of coevolutionary tournaments $T(VS(n))$. In this article, we state and show that a coevolutionary problem is either solvable (its corresponding tournament is reducible) or not (the tournament is irreducible).

A coevolutionary tournament $T(VS(n),A)\u2208T(VS(n))$ on $n\u22653$ is either reducible or irreducible.

This is simply a direct consequence of the definition of reducibility and irreducibility of tournaments (as given by Definition ^{19}).^{6}

$\u25a1$

We next show that a solvable coevolutionary problem has a subset of solutions that are preferred over the remaining solutions.

A reducible coevolutionary tournament $T(VS(n),A)\u2208T(VS(n))$ of order $n\u22652$ can be vertex-partitioned into two disjoint nonempty subsets $VS(n)1$ and $VS(n)2$. The dominant subset $VS(n)2$ contains solutions that dominate all remaining solutions in the other subset $VS(n)1=VS(n)-VS(n)2$. The minimal dominant subset contains a single solution.

The first part of the lemma is simply a direct consequence of the definition of reducibility of tournaments. It remains to prove the second part. We first make note that every tournament can be decomposed into subtournaments $Ti(i=1,2,3,\u2026,l)$ whereby (Moon, 1968):

Every vertex in $T(j)$ dominates every vertex in $T(i)$ if $1\u2264i<j\u2264l$.

Every subtournament $T(i)$ is either irreducible or transitive.

No two consecutive subtournaments $T(i)$ and $T(i+1)$ are both transitive.

Consider the trivial case whereby the tournament is transitive, $l=1$. As a consequence of Theorem ^{23}, a tournament $T(VS(n),A)\u2208T(VS(n))$ of order $n>2$ is transitive if and only if it is acyclic. Obviously, we can always vertex partition a transitive tournament by taking the vertex with in-degree $n-1$ as $VS(n)2$ and all the remaining $n-1$ vertices as $VS(n)1$ so that $VS(n)1\u21d2VS(n)2$. For a reducible tournament, assume that it can be decomposed into two subtournaments ($l=2$). Since the two subtournaments $T(1)$ and $T(2)$ cannot be both transitive, then one of them must be irreducible. Let $V(T(1))\u21d2V(T(2))$. $V(T(2))$ with a single vertex is by definition strong and as such irreducible.

$\u25a1$

We need to show the case above only where the dominant subset is minimal. There are other cases as well; for example, $T(2)$ is transitive and $T(1)$ is irreducible. Here, even though $V(T(1))\u21d2V(T(2))$, the single dominant vertex in a transitive $T(2)$ obviously dominates all other vertices in $T(2)$ and $T(1)$ as well. In another case, we have both irreducible subtournaments. For these coevolutionary tournaments, the maximal dominant subset that induces $T(2)$ and for which $V(T(1))\u21d2V(T(2))$ is of order $n-1$.

For coevolutionary problems that are not solvable, we have the following result.

An irreducible coevolutionary tournament $T(VS(n),A)\u2208T(VS(n))$ is strongly connected. It is Hamiltonian and furthermore, vertex-pancyclic.

This is simply a direct consequence of Theorems ^{21}, ^{22}, and ^{23}.

$\u25a1$

The main implication is that an irreducible coevolutionary tournament does not have a subset of dominant solutions. In addition to every $u,v,w\u2208VS(n)(T)$ contained in a 3-cycle, this coevolutionary tournament is vertex-pancyclic. One can find the solution $vn$ that dominates all other solutions $v2,v3,v4,\u2026,vn-1$ but is dominated by $v1$, which in turn is dominated by $v2,v3,v4,\u2026,vn-1$ (see Figure 1). This is a consequence of the Hamilton cycle in the tournament. In the context of a game and optimality of a strategy that is defined in terms of the number of opponent strategies it defeats, one finds the situation whereby the *best strategy defeats all but the worst strategy*. The lack of dominant solutions becomes more apparent in the case of *regular* coevolutionary tournaments whereby each solution $v$ has the same score with $dD-(v)=dD+(v)$, which generalize $T(VS(3),A)$ with $u\u2192v\u2192w\u2192u$ (as in the game *rock–paper–scissors*).

Our results can be extended to the family of coevolutionary semicomplete digraphs $SD(VS)$ as well. The arguments behind Theorems ^{21}, ^{22}, and ^{23} for tournaments can also be made for their simple generalization in the form of semicomplete digraphs. For example, Theorem ^{23} on strong connectivity and vertex pancyclicity for tournaments has been restated for semicomplete digraphs in Bang-Jensen et al. (1997). The original argument (Moon, 1968) that makes use of a proof by induction of a $k$-cycle (where $3\u2264k\u2264n)$ containing a vertex $v$ and the existence of a $(k+1)$-cycle containing $v$ remains unchanged if one replaces instances of tournaments with semicomplete digraphs. One can make an observation (e.g., see Figures 1a and 1b) that the introduction of 2-cycles does not make a semicomplete digraph vertex pancyclic. Instead, it arises from the presence of 3-cycles and the Hamilton cycle.

Unlike the case of irreducible coevolutionary digraphs, one can employ search procedures to discover the dominant subset in a reducible coevolutionary digraph. We first introduce the following notion on strong connectivity of subdigraphs $D(i)$ of $D$ (Bang-Jensen and Gutin, 2009). Then, we conclude with a useful description of structural properties in reducible coevolutionary digraphs that we apply later.

The *strong component* of $D$ is a maximal induced strong subdigraph of $D$. It is maximal in the sense that if a strong $D(i)$ and strong $D(j)$ are connected by arcs in both directions, in which case there exists both $(u,v)$-path and $(v,u)$-path for every $u\u2208D(i)$ and $v\u2208D(j)$, then they belong to the same strong subdigraph $D(k)=D(i)\u222aD(j)$. Obviously a single-vertex $D(i)$ is strong by definition. A digraph $D$ can be decomposed into its strong components $D(1),D(2),D(3),\u2026,D(l)$, with each strong component being distinct by definition of it being a maximal induced subdigraph.

A reducible coevolutionary semicomplete digraph $SD(VS(n),A)\u2208SD(VS(n))$ of order $n\u22652$ has a strong decomposition $VD(1)\u222aVD(2)\u222aVD(3)\u222a\u2026\u222aVD(l)$ with a unique ordering $D(1),D(2),D(3),\u2026,D(l)$ whereby $D(i)\u21d2D(j)$ when $i<j$. $D(1)$$(D(l))$ is the initial (terminal) strong component. In particular, the dominant solutions of $SD(VS(n),A)$ are contained in its terminal strong component.

As with the case of tournaments, we first note that a reducible semicomplete digraph can be decomposed into strong components $Di(i=1,2,3,\u2026,l)$. Let the strong components be ordered as $D(1)\u21d2D(2)\u21d2D(3)\u21d2\u2026\u21d2D(l)$ with $D(i)\u21d2D(j)$ when $i<j$ (Bang-Jensen et al., 1996). One can perform a *contraction* operation on the strong components that replaces $D(i)$ with a single vertex $gD(i)$ and remove all parallel arcs (Bang-Jensen and Gutin, 2009). When contraction is applied to all the strong components of $D$, the resulting digraph is called the *strong component digraph*$SC(D)$. Note that $SC(D)$ is now $gD(1)\u2192gD(2)\u2192gD(3)\u2192\cdots \u2192gD(l)$ with $gD(i)\u2192gD(j)$ when $i<j$; that is, it has an acyclic ordering. The *initial* (*terminal*) strong component of $D$ is obviously the strong component whose corresponding vertex in $SC(D)$ has zero in-degree (out-degree). The same argument can be made for tournaments. However, with a reducible tournament one may have to apply contraction on transitive subtournaments, in which case the resulting digraph will be acyclic as well.

$\u25a1$

### 2.8 Quantitative Characterizations of Coevolutionary Digraphs

In demonstrating several structural results relating to cycle structures in coevolutionary digraphs crucial to characterizing their dichotomy in solvability, we make no assumption on availability of prior information relevant to uncovering those structures. Where more information becomes available, we can make quantitative characterizations relevant to the *complexity* of those cycle structures and further uncover their higher organizations. In particular, we first study how those cycles of various lengths can be *packed* in a manner that results in a strong decomposition of the coevolutionary tournament. We present the following result.

The first statement on score sequences of coevolutionary tournaments is a direct consequence of a theorem by Landau (1953) (Bang-Jensen and Gutin, 2009).^{7} The second statement on score sequences of irreducible coevolutionary tournament is a direct consequence of a theorem by Moser and Harary (1966) (Bang-Jensen and Gutin, 2009). The final statement immediately follows from the second statement, thereby providing conditions for reducibility of coevolutionary tournaments (Moon, 1963).

$\u25a1$

Lemma ^{29} makes the only assumption of a labeled tournament (its score sequences are known). There are several consequences to this result. A coevolutionary tournament of order $n$ is transitive if it has a score sequence of $(0,1,2,\u2026,n-1)$ whereby $\u2211i=1ksi=k2$ for each $k$, $1\u2264k\u2264n$. Lemma ^{29} can be used to identify strong components in a reducible coevolutionary tournament. The idea is that to look for successive values of $k$ for which the equality $\u2211i=1ksi=k2$ holds. This follows from Lemma ^{25} on the decomposition of tournaments into subtournaments $Tj(j=1,2,3,\u2026,l)$. A strong subtournament $T(j)$ will be sandwiched between strong or transitive subtournaments, $T(j-1)\u21d2T(j)\u21d2T(j+1)$.

One can formalize this as follows. We consider a coevolutionary tournament $T(VS(n),A)\u2208T(VS(n))$ with its solution set $VS(n)$ indexed according to its score sequence. Then, let $\u2211i=1psi=p2$, $\u2211i=1qsi=q2$, and $\u2211i=1ksi>k2$ for $p+1\u2264k\u2264q-1$ with $0\u2264p<q\u2264n$. The subtournament induced by the subset ${vp+1,vp+2,vp+3,\u2026,vq}$ is a strong subtournament $T(j)$. Furthermore, the score sequence that is restricted to the strong subtournament $T(j)$ itself is given by $(sp+1-p,sp+2-p,sp+3-p,\u2026,sq-p)$ (Pirzada, 2012). If $T(VS(n),A)$ is irreducible, the procedure will just return a single strong component—the entire tournament itself.

Finally, we will exploit Lemma ^{29} to construct an index relevant to characterizing complexity of cycle structures in coevolutionary digraphs. We give a precise definition for the index as follows.

The index $\nu $ ranges from 1 to $n$. Essentially, it is a count of the number of strong components in the coevolutionary tournament. When this index is used together with structural results such as Lemma ^{28} that provide qualitative descriptions of tournament decomposition, it provides a useful macroscale indication of cycle structure complexity in coevolutionary tournaments that is summarized as a simple quantity. $\nu =1$ if and only if the tournament is strong, while $\nu =n$ if and only if the tournament is transitive (each vertex is a strong component and all these vertices have an acyclic ordering). A value of $\nu $ closer to $n$ indicates the presence of many shorter length cycles that can be grouped separately into different subtournaments. When $\nu $ is close to one, the tournament has longer length cycles between $n/2$ to $n-1$. $\nu $ can be computed quickly (the computation cost is linear with the order of the tournament with precomputed $L$s). Later, we demonstrate how this index is used in computational studies.

## 3 Coevolutionary Processes on Digraphs

We introduce the second part of our framework and show that coevolutionary processes can be formulated naturally as sampling processes on digraphs. For simplicity and to allow for in-depth analysis, we consider population-one coevolutionary systems, which corresponds to random walks on digraphs. These would cover various coevolutionary systems such as those that are used in the context of learning game-playing strategies through self play. We will formulate these random walks as discrete time Markov chains operating on coevolutionary digraphs. This will allow us to study how specific structures in coevolutionary digraphs can affect the dynamics of coevolutionary processes.

### 3.1 Coevolutionary Processes as Random Walks on Digraphs

Algorithm 1 describes a standard random walk on labeled digraphs $D$. It starts at a random vertex and in each subsequent step, randomly moves to an out-neighbour of the current vertex. This requires prior information on connectivity structures (out-neighbourhoods of all vertices in $D$). In practice, one would implement a random walk on unlabeled digraphs such as the (1+1)CEA (Algorithm 2). In the (1+1)CEA, the random move in each step is made through a two-stage process of variation (generates a random *offspring* from the neighbourhood of the *parent* vertex) and selection (moves to the offspring vertex if it dominates the parent vertex). Unlike a random walk, there is a chance the (1+1)CEA stays at the current vertex in a step because it chooses one of its in-neighbours (the probability is proportionate to the order of the in-neighbourhood). Note as well that in practice a (1+1)CEA can apply an arbitrary stopping criterion (finite number of steps as in Algorithm 2).

### 3.2 Formulating the Coevolutionary Markov Chain

Regardless of how random walks on digraphs are implemented algorithmically, our focus is on the underlying *stochastic processes* that generate those walks. We can exploit various tools for quantitative analysis of coevolutionary processes consistent with the qualitative characterizations we have presented earlier on coevolutionary problems. We first motivate formulating coevolutionary processes on digraphs as a specific class of stochastic processes—discrete time Markov chains. We will refer readers to Norris (1998) as an introductory text and Meyn and Tweedie (2009) for more advanced materials on Markov chains. We will provide relevant materials to ensure this article is self-contained.

A *discrete time stochastic process*$\Phi $ is a collection of ${\Phi t:t\u2208N0}$ of random variables with each $\Phi t$ taking values in a countable set $X$ at time $t$. $N0$ is the set of natural numbers that include 0. A *random walk* is a discrete time process $\Phi ={\Phi t:t\u2208N0}$ that takes a step on the digraph from $\Phi t$ to $\Phi t+1$. Each $\Phi t$ takes values from the *state space*$X$ corresponding to the finite set of vertices $V(D)$ of the digraph $D=(V,A)$. Each state $x\u2208X$ is the distinct vertex $v\u2208V$.

Constructing coevolution as a discrete time process is technically involved as one needs to work on probability spaces. In general, one first prescribes the structural definition for the sample path behaviors and the dynamics of steps on the digraph through time to define the probability laws that govern the evolution of the chain $\Phi $. Even in the usual case where sequences ${\Phi 0,\Phi 1,\Phi 2,\u2026}$ take *discrete* values in $X$, developing a complete, functional description of the distribution of the random walk on most digraphs other than those with structures that strictly constrain its sample path behaviors (e.g., acyclic tournaments) is difficult. Instead, we use the observation that coevolutionary processes satisfy the *Markov property*—the step to a future state only depends on the current state.^{8} We exploit this to define and construct the probabilistic motion of the coevolutionary process on digraphs.

We first note the following definition and facts of a Markov chain.

^{9}

*time homogeneity*(Theorem 1.1.1, page 2 in Norris, 1998) whereby

*memory loss*(Theorem 1.1.2, page 3 in Norris, 1998) whereby the probabilities $P(xt,xt+1)$ are independent of the time step $t$.

We formulate and define the Markov chain for a coevolutionary process on digraphs as follows.

Let $D(VS(n),A)\u2208D(VS(n))$ be a coevolutionary digraph having a finite vertex set $VS(n)$. A coevolutionary Markov chain is a random walk on $D(VS(n),A)$ with initial distribution $\mu $ over and Markov transition matrix $P$ on the state space $X=VS(n)$ satisfying

$\mu =(\mu x:x\u2208X)$, where $0\u2264\mu x\u22641$ and $\u2211x\u2208X\mu x=1$, and

$P=(P(x,z):x,z\u2208X)$, where for every row is a distribution whereby $0\u2264P(x,z)\u22641$ and $\u2211y\u2208XP(x,y)=1$.

Guarantees that such a *coevolutionary Markov chain*$\Phi ={\Phi t:t\u2208N0}$ defined on the state space $X=VS(n)$, initial distribution $\mu $ and transition matrix $P$ are given by the *Existence Theorem* of Markov chains on countable state space (Chung, 1960). By definition a coevolutionary digraph $D(VS(n),A)$ has a discrete $VS(n)$ with finite number of elements, which is a countable set.

*one-step transition probabilities*$P(xt,xt+1)$ (Meyn and Tweedie, 2009). Other useful computations can be made, for example, the probabilities starting from $x$ after $n$ steps that the chain is at $y$, $P\mu (\Phi n=y|\Phi 0=x)=Pn(x,y)$. One first sets $P0=I$ (the identity matrix), which corresponds to the initial conditions $P0(x,x)=1$ for all $x\u2208X$. Then, the

*n-step transition matrix*given by the iterates $Pn=(Pn(x,y):x,y\u2208X)$ for $n\u22650$ can be defined inductively through $Pn+1=PnP$ using the following expression

### 3.3 Coevolutionary Markov Chain Examples

We construct four coevolutionary Markov chains (two for each case of transitive and regular tournaments). The initial distribution can be arbitrary so we just need to construct the appropriate Markov transition matrix. These examples highlight cases where we can do the constructions formally and later in Section 4.4 to perform quantitative analysis without resorting to sampling and large-scale computations as is often the case in real-world problems. The first example is the random walk on labeled transitive coevolutionary tournaments. Let $T(VS(m),A)$ be a transitive coevolutionary tournament with vertices $(v1,v2,\u2026,vm)$ indexed according to its score sequence $(s1,s2,\u2026,sm)$. We simplify our notation and let $Pm\xd7m=(pij:i,j\u2208{1,2,\u2026,m})$ be the Markov transition matrix for this coevolutionary Markov chain on $T(VS(m),A)$.

*adjacency matrix*$Madj$ can be constructed as follows (Brualdi, 2010):

^{10}

*domination matrix*$Mdom$ where $dom(i,j)=1$ means $i\u2190j$. Note that $Mdom$ is the transpose of $Madj$, $Mdom=MadjT$. We can then compute the Markov transition matrix for a random walk on labeled regular coevolutionary tournament as

## 4 Quantitative Characterizations of Coevolutionary Markov Chains

In this section, we first demonstrate the link between structural properties in the coevolutionary digraphs and their impact on coevolutionary processes that operate on those digraphs. Subsequently, we will introduce and develop quantitative characterizations that are relevant to those coevolutionary Markov chains.

### 4.1 Class Structures

One fundamental notion that can help us understand the dynamics of a coevolutionary Markov chain is the *communication* classes on its state space. We can analyze a large and complex coevolutionary Markov chain by breaking it into smaller pieces having shared properties (states within them are equivalent in some sense). Let $D(VS(n),A)$ be a coevolutionary digraph. $P$ is the Markov transition matrix of the coevolutionary Markov chain operating on $D(VS(n),A)$. Let $x$ and $y$ be two distinct states $x,y\u2208X=VS(n)$.

$x\u21ddy$ ($x$*leads to*$y$) if and only if there is a positive probability that starting at $x$ the Markov chain reaches $y$ after $n$-steps, $pxy(n)>0$ for some $n\u22650$.

The relation $\u21dd$ is transitive given the property of the $n$-step transition probability (as indicated by Equation 4) so that one can compute $pxz(m+n)\u2265pxy(m)pyz(n)>0$ for some $m\u22650$ and $n\u22650$. By definition, $x\u21ddx$, in which case the relation is reflexive.

$x\u21ady$ ($x$*communicates* with $y$) if and only if $x\u21ddy$ and $y\u21ddx$.

In this case, the relation is symmetric and transitive. As such, the relation $\u21ad$ forms an equivalence relation on $X$, and allows one to partition $X$ into disjoint subsets that are called communicating classes (or equivalence classes) (Chung, 1960).

Let $C(x)$ be the class containing $x\u2208X$. A state space $X$ consisting of a single class $C(x)=X$ (and the Markov chain operating on $X$) is said to be *irreducible*.

There are intimate relationships between the class structure of a coevolutionary Markov chain and the connectivity structure of the coevolutionary digraph that the chain operates on. One obvious connection is the notion of irreducibility.

A coevolutionary Markov chain is irreducible if it operates on an irreducible coevolutionary digraph.

An irreducible digraph $D$ is strongly connected whereby for every $x,y\u2208V(D)$, there is an $(x,y)$-path and $(y,x)$-path. This implies that $x\u21ady$ for all $x,y\u2208X$ for a coevolutionary Markov chain operating on $X=V(D)$. There is only a single communicating class $C(x)=X$, in which case the coevolutionary Markov chain is irreducible.

$\u25a1$

States can be classified as *essential* or *inessential*. A state $x$ is essential if and only if for all $y$, $x\u21ddy$ implies $y\u21ddx$. Otherwise, $x$ is inessential when there is a $y$ such that $x\u21ddy$ but $y\xac\u21ddx$.

The property that a state is essential or inessential is a *class property* (all the states in a communicating class share the same property). An essential state cannot lead to an inessential state. Crucially, an essential class is by definition *minimal closed* – it is closed in the sense that $\u2200y\u2208C(x)P(y,C(x))=1$ and minimal in that it does not contain a proper subset that is closed (Chung, 1960). In other words, a coevolutionary Markov chain will be *absorbed* into an essential class upon entering it (i.e., it will be unable to leave it once it first enters it). This leads to the following lemma.

For a reducible coevolutionary Markov chain, there is only one absorbing class. The absorbing class consists of states corresponding to dominant solutions in the coevolutionary digraph.

A coevolutionary Markov chain operating on a reducible coevolutionary digraph is reducible. By Lemma ^{28}, a reducible coevolutionary semicomplete digraph will have a terminal strong component that dominates all other remaining vertices. States corresponding to the terminal strong component form an essential communication class $C(x)={y:x\u21ady}$ that is also absorbing.

$\u25a1$

Consider a reducible coevolutionary Markov chain with Markov transition matrix $P$ having an absorbing class $C$. Let $PC$ be the matrix restricted to states in $C$. Then, there exists an irreducible coevolutionary Markov chain $\Phi C$ with state space restricted to $C$ and a Markov transition matrix given by $PC$.

This is a direct consequence of Proposition 4.1.2 (page 79 in Meyn and Tweedie, 2009).

$\u25a1$

It remains then to investigate the behavior of the coevolutionary Markov chain on $B$ and the manner in which it gets absorbed to $C$. The next section will provide the means for which this can be done quantitatively.

### 4.2 Hitting Times

The analysis of the behaviors of the chain $\Phi $ is often concerned with its distributions at certain *random times* as it evolves on $X$. Additional tools are needed for which these behaviors can be described in a precise and quantitative manner. One of the quantities refers to the random times with which $\Phi $ visits certain states in $X$.

*first hitting time*of $A$ is the random variable $\tau A:X\u2192N0\u222a{\u221e}$, which is given by

One can also define the first visit to $A$ to only consider positive times $\tau A+=inf{t\u22651:\Phi t\u2208A}$ (Meyn and Tweedie, 2009).

*first return time*is defined as

Obviously, $\u03f1A$ is the case of $\tau A+$ with $\Phi 0\u2208A$.^{11} Another quantity relates to the number of visits to some states.

*occupation time*$\eta A$ is defined as

The occupation time $\eta A$ counts the number of visits of a coevolutionary Markov chain $\Phi $ to $A$ after time zero. It is crucial in the following characterization of the set $A$.

$A$ is uniformly *transient* if there is a finite $Ex(\eta A)$ for all $x\u2208A$. Otherwise, $A$ is *recurrent* if $Ex(\eta A)=\u221e$ for all $x\u2208A$.

This can be extended to individual states as well, i.e., a state is recurrent if $Ex(\eta x)=\u221e$ (Meyn and Tweedie, 2009). This leads to the following lemma.

Consider an irreducible coevolutionary Markov chain $\Phi $ on $X$. Every state $x\u2208X$ is recurrent.

From Lemma ^{37}, an irreducible $\Phi $ operates on $X$ that coincides with an irreducible coevolutionary digraph $D=(V,A)$. Such a digraph is Hamiltonian from Lemma ^{26}, in which case there is a closed finite-length $(x,x)$-path for all $x\u2208V(D)$. This implies that $Px(\u03f1A<\u221e)=1$ for all $x\u2208X$. Applying Proposition 8.1.3 in Meyn and Tweedie (2009), we have that $Ex(\eta x)=\u221e$. As such, all $x$ is recurrent.

$\u25a1$

How about the case of a reducible coevolutionary Markov chain? Obviously, its state space can be partitioned into two classes—a closed absorbing class of recurrent states and another consisting of transient states only. As in the context of analysis of *Evolutionary Algorithms* (EAs) (He and Yao, 2003), studying the expected (mean) hitting time of an absorbing coevolutionary Markov chain $\Phi $ is crucial. Unlike EAs, complete connectivities in the coevolutionary digraph means $\tau A+=1$ although it is a rare event for $\Phi $ to make a one-step walk to $A$. The following gives a precise definition for this quantity (Norris, 1998).

*canonical form*of its Markov transition matrix, $P*$ (Meyer, 2000):

This is a direct consequence of Theorem 3.2 in Iosifescu (1980) for absorbing Markov chains. In our case, it follows from Lemma ^{39} that a reducible coevolutionary Markov chain is an absorbing Markov chain with one absorbing class.

$\u25a1$

*fundamental matrix*of the Markov transition matrix $P$. One can perform direct computation of the expected hitting time of any reducible coevolutionary Markov chain by first computing its fundamental matrix (Theorem 11.4 in Grinstead and Snell, 1997):

### 4.3 Invariant Measures—Stationary Distribution

This section is concerned with characterizing the coevolutionary Markov chain $\Phi $ in the context of the long-time properties of $\Phi $. This is relevant for irreducible coevolutionary Markov chains as well as reducible chains (after the chain enters the absorbing class). We first introduce the notion of an *invariant measure* (Norris, 1998).

*invariant*if

A $\pi $ that is defined on $X$ and satisfies the invariant property would represent the long-term limiting distribution of the coevolutionary Markov chain. In addition, such a distribution can be described in the following equivalent terms:

*Stationary*—If $(\Phi n)n\u22650$ is a coevolutionary Markov chain with $\mu =\pi $ and $P$, then $(\Phi m+n)n\u22650$ having $P(\Phi m=x)=(\pi Pm)x=\pi x$ for all $x\u2208X$ is also a coevolutionary Markov chain with $\mu =\pi $ and $P$ (Theorem 1.7.1 in Norris, 1998).*Equilibrium*—For all $x\u2208X$,^{12}$Pn(x,y)\u2192\pi y$ as $n\u2192\u221e$ for all $y\u2208X$ (Theorem 1.7.2 in Norris, 1998).

It is known that for an irreducible Markov chain $\Phi $ on countable state space $X$, its stationary distribution can be calculated from the *expected return times*$Ex(\u03f1x)$ (Levin et al., 2009). The general intuition is that if one breaks up the trajectory of $\Phi $ into identically distributed segments through visits to and from some arbitrary states, the average fraction of time that is spent on a state $x\u2208X$ by each segment would be similar to the long-term fraction of time spent on $x$ by $\Phi $. Furthermore, starting from any arbitrary distribution $\mu $, the chain would converge to its stationary distribution in the long run (Norris, 1998; Levin et al., 2009). For coevolutionary Markov chains, we have the following results.

Consider an irreducible coevolutionary Markov chain with Markov transition matrix $P$. Then, there is a unique invariant probability distribution $\pi =(\pi x:x\u2208X)$ on $X$ where $0\u2264\pi x\u22641$ and $\u2211x\u2208X\pi x=1$ such that $\pi P=\pi $. Furthermore, $\pi x=1/Ex(\u03f1x)$.

This is a direct consequence of Theorems 1.7.7 in Norris (1998) whereby an irreducible and positive recurrent Markov chain on a finite state space has a unique invariant probability measure $\pi =(\pi x:x\u2208X)$ on $X$ with $\pi x=1/Ex(\u03f1x)$. A state $x$ is positive recurrent if its expected return time $Ex(\u03f1x)$ is finite. By Lemma ^{37}, obviously all $x\u2208X$s are positive recurrent since $Ex(\u03f1x)$ is finite.

$\u25a1$

Consider a reducible coevolutionary Markov chain with Markov transition matrix $P$. For all inessential states $y\u2208X$, $\pi y=0$. Then, there is a unique invariant probability distribution $\pi $ concentrated on the absorbing class $C$ of essential states in $X$.

Let $B$ be the set of all inessential states $y\u2208X$. The first statement on $\pi y=0$ for all inessential states $y$ is a direct consequence of Proposition 1.25 (page 16 in Levin et al., 2009). This implies that the probability distribution is concentrated on the remaining essential states of the absorbing class $C$. From Lemma ^{40}, we can study the irreducible coevolutionary Markov chain with $PC$ separately and obtain $\pi C=(\pi x:x\u2208C(x))$ as a result of Lemma ^{49}. Since $X=C\u222aB$ and $C\u2229B=\u2205$, then $\pi =(\pi C,\pi B)$ whereby all elements in $\pi B$ are zero.

$\u25a1$

This is a direct consequence of the *Convergence to Equilibrium Theorem* (given as Theorem 1.8.3 in Norris (1998). We just need to show that an irreducible coevolutionary Markov chain $\Phi $ is aperiodic. By Lemma ^{26}, irreducible coevolutionary digraphs are vertex pancyclic, in which case the greatest common divisor to the cycle lengths is one. As such, $\Phi $ is aperiodic.

$\u25a1$

### 4.4 Case Studies on Quantitative Measures of Coevolutionary Markov Chains

We first provide analytic expression for the expected hitting time of specific coevolutionary Markov chains. This case study considers both the random walk and (1+1)CEA on transitive coevolutionary tournaments. We note that the Markov transition matrices given by Equations 5 and 6 are in the form of an upper triangle matrix. It requires re-labeling the indices in $P$ so that it becomes a lower triangle matrix. For these two cases, $Q$s are also in the form of a lower triangle matrix. Crucially, each of the $n\xd7n$ matrices $(I-Q)$ has a $rank(I-Q)=n$ so its inverse exists (Meyer, 2000).

*Gauss-Jordan*method to obtain the $m\xd7m$ fundamental matrix, and then use this result to partially fill the $m\xd7m$ submatrix of the $(m+1)\xd7(m+1)$ fundamental matrix. Starting from $m=2$ and applying this iteratively, we obtain the following form:

^{47}to calculate the expected hitting time and then obtain

In this second part of our case study, we calculate the stationary distribution of a specific class of coevolutionary Markov chain—those operating on regular tournaments. We can make use of structural results from digraph theory to obtain a general result on the stationary distribution of regular coevolutionary Markov chains.

Consider a coevolutionary Markov chain operating on a regular coevolutionary tournament $T(VS(n),A)\u2208T(VS(n))$ of odd order $n\u2265(3+2m)$, $m=0,1,2,\u2026$. This regular coevolutionary Markov chain has a unique invariant probability measure $\pi =(\pi x:x\u2208X)$ on $X=VS(n)$ whereby $\pi x=1/n$ for all $x$.

Every regular coevolutionary tournament is *arc-pancyclic*, which is a direct consequence of Corollary 6.6.4 in Bang-Jensen and Gutin (2009). An arc-pancyclic coevolutionary tournament is arc-$k$-cyclic for all $k={3,4,5,\u2026,n}$; that is, each arc of $T(VS(n),A)$ is contained in a cycle for each length of $k={3,4,5,\u2026,n}$. Since $dD+(x)=dD-(x)=(n-1)/2$ for all $x$ in a regular $T(VS(n),A)$, this means that there is an equal number of arcs of various lengths $k={3,4,5,\u2026,n}$ that start and end at each $x\u2208X$. Any random walk on $T(VS(n),A)$ would thus spend equal fraction of time for each $x$, in which case $\pi x=1/n$ for a tournament with $n$ vertices.

$\u25a1$

One immediate consequence of Lemma ^{52} is that any coevolutionary process (random walk or (1+1)CEA for example) can be defined on a regular tournament of order $n$ and that $\pi =(1/n,1/n,1/n,\u2026,1/n)$ since the cycles that start from and return to each vertex are the same. We can also apply this result with Lemma ^{50} to calculate the stationary distribution of a reducible coevolutionary Markov chain having an absorbing class that is essentially a regular subtournament.

#### 4.4.1 Absorbing Coevolutionary Markov Chains

Earlier result for (1+1)CEA with uniform expected hitting time for any starting vertex indicate unique properties (the diagonal elements in the probability transition matrix), which we can generalize further with the following result.

Consider a (1+1)CEA operating on a reducible coevolutionary tournament $T(VS(n+nd),A)\u2208T(VS(n+nd))$. Let $VS(n+nd)$ consists of two disjoint subsets $VS(n)1$ and $VS(nd)2$, whereby $VS(n)1\u2229VS(nd)2=\u2205$. Let $VS(nd)2\u21d0VS(n)1$. Let the coevolutionary Markov chain be defined with Markov transition matrix in its canonical form $P*$. Let $i\u2208Q$ be the transient states and that $|Q|=n$. Let the column vector $h=[hi]i\u2208Q$ where $hi=Ei(\tau A)$ is the expected hitting time starting from a transient state $i$ to the absorbing class $A=VS(nd)2$. Then, $h=[hi]i\u2208Q=[n,n,n,\u2026,n]T$.

^{28}that the absorbing class $A=VS(nd)2$ must be a strong component. One can perform a contraction operation on $VS(nd)2$ and replace it with a single vertex and remove all parallel arcs. The resulting coevolutionary tournament is $T(VS(n+1),A)$. The absorbing class $A$ is now a single absorbing state and that

For every transient state $i\u2208Q$, note that the corresponding vertex $i$ in the co-evolutionary tournament $T(VS(n+1),A)$ has the same degree $dD(i)=dD+(i)+dD-(i)=(n+1)-1=n$. Let $R=(ri:i=1,2,3,\u2026,n)$ and $Q=(qij:i,j=1,2,3,\u2026,n)$. Given our definition of (1+1)CEA (as in Algorithm 2), for each $i$: (1) $qii=dD-(i)dD(i)$ and (2) $(\u2211j\u2208ND+(i)\u2216xaqij)+ri=dD+(i)dD(i)$. The one-step transition probability from $i$ to the single absorbing state $xa$ is given by $ri$. Since each one-step transition from $i$ to its outneighbour $j$ occurs with equal probability, then $\u2200j\u2208ND+(i)qij=1dD(i)$ and $ri=1dD(i)$. Note that $\u2200j\u2208ND-(i)qij=0$. As such, each row of $R1+Q1$ sums to one, i.e., $\u2200i(ri+\u2211j=1nqij)=dD(i)dD(i)=1$.

Next, we need to show that the corresponding row sums of $(I-Q)-1=I+Q+Q2+Q3+\cdots $ follow a geometric series. We need to prove that $Qn\xd7nm1=[(n-1n)m,(n-1n)m,(n-1n)m,\u2026,(n-1n)m]T$. This will give the expected hitting times calculated from the geometric series $1+n-1n+(n-1n)2+(n-1n)3+\cdots =1/(1-n-1n)=n$. We do this by induction. Let the column vector $Qn\xd7nm1=[Qm]i=1n$, with $[Qm]i$ representing the $i$th element of $[Qm]i=1n$. The induction hypothesis is that $Qn\xd7nm+11=[QmQ1]i=1n$.

Basis ($m=0$): By definition $Qn\xd7n0=In\xd7n$. Then, $Qn\xd7n01=In\xd7n1=1$.

Inductive step: We need to show that $Qn\xd7nm+11=[QmQ1]i=1n$. We first note that $Qm+11=QmQ11$. Let $Qn\xd7nm=(xij:i,j=1,2,3,\u2026,n)$ and $Qn\xd7n1=(yij:i,j=1,2,3,\u2026,n)$. Let $Qn\xd7nmQn\xd7n1=(zij:i,j=1,2,3,\u2026,n)$. $Qim+1=\u2211j=1nzij$. Since $\u2200iri=1dD(i)=1n$, then $\u2200iQi1=1-1n=n-1n$.

From our induction hypothesis, it can be seen that $Qn\xd7nm1=[Qm]i=1n=[(nn+1)m]i=1n$. Each row sum of $h=(I-Q)-11=(I+Q+Q2+Q3+\cdots )1$ then follows the geometric series $1+n-1n+(n-1n)2+(n-1n)3+\cdots =1/(1-n-1n)=n$. Thus, $h=[hi]i\u2208Q=[n,n,n,\u2026,n]T$.

$\u25a1$

The procedure to show that the row sums of $(I-Q)-1=I+Q+Q2+Q3+\cdots $ follow a geometric series is the same as in Theorem ^{53}. It can be shown that $Qn\xd7nm1=[Qm]i=1n=[(n-1n)m]i=1n$. Each element of $h=(I-Q)-11$ then follows the geometric series $1+n-1n+(n-1n)2+(n-1n)3+\cdots =1/(1-n-1n)=n$, in which case, $h=[hi]i\u2208Q=[n,n,n,\u2026,n]T$.

### 4.5 Generating Random Coevolutionary Tournaments

We conclude this section with a computational study on generating random coevolutionary tournaments in a controlled manner. Potential applications include using random tournaments as benchmarks to study methodologies estimating quantities of interest in more complex and realistic coevolutionary Markov chains. We first note that the simple approach of randomly orienting edges of the underlying graphs would usually generate irreducible random tournaments (Moon, 1968). There is more than $50%$ chance to generate an irreducible random tournament of order five, which increases to more than $99.9%$ for random tournaments of order 16! Beyond that, generated random tournaments would almost certainly be irreducible. Another approach is to first generate all valid score sequences (Hemasinha, 2003), which are used to reconstruct the corresponding tournaments (Moon, 1968).

This setting can be motivated by statistical physics. Each node of the existing tournament has an ``*energy*'' related to its score (in-degree). The new incoming node $n+1$ will be more likely to orient edges towards high energy node in the existing tournament. Parameter $\beta $ plays the role of inverse temperature. For high temperature regimes, the energy of nodes begins to equalize and the probability of edge orientation is the same for any node in the existing tournament. In contrast, by cooling the system (increasing $\beta $), orientation of edges from the new node $n+1$ towards high energy nodes of the existing tournament becomes more persistent.

We have performed initial experiments to determine appropriate choices for the parameters involved. We initialize (seed) the graph growth with transitive tournaments. For the statistics ${xi}1n$, we use the corresponding node ranks $(i)1n$ with $x\xaf=n/2$, $\beta \u2208[0,4]$. Tournaments of order $nfinal=1000$ are generated from initial ones of order $ninit\u2208{10,100}$. Each experiment set is repeated 100 times, thus generating 100 tournaments randomly and independently. The results are summarized in Figure 3. Obviously, the number of strong components are higher when the initial tournament seeds are larger as shown in Figure 3b.

We next introduce into our 1000-node tournaments an additional single dominant (1001-th) node that has all the edges from the original 1000 vertices oriented towards it. This construction makes the tournaments reducible and so we can compute the expected hitting time from the *weakest* node $x1$ (with the lowest score) to the dominant (absorbing) node $x1001$. We now study the impact of increasing transitive structures, measured by the number of strong components (reflected in the complexity (Landau's) index $\nu $ (Equation 1)), on dynamics of the coevolutionary process, measured by the expected hitting times of the dominant node.

First, we show results for the experiment that uses the standard random walk (Algorithm 1) with initial seed tournament of order 10. We can clearly see signatures of higher and lower temperature regimes in Figure 4a. In the high temperature regime ($\beta \u2248[0,0.001]$), all the base 1000-node tournaments are irreducible, hence there is only one strong component. Together with the dominant 1001-th node, there are two strong components. There is little variability of hitting times (around 492–497), which reflects limited variability of the internal structure of the irreducible 1000-node base tournaments. For comparison, the expected hitting time is exactly 500 for a regular 999-node base tournament. In the low temperature regime ($\beta \u2248[1,4]$), prominent transitive structure dominates the base tournaments and the hitting times range in $[7.486,10.659]$. For comparison, the expected hitting times would be slightly less than 10 for a vertex pancyclic 1000-node base tournament (which has the longest transitive structure) and around 7.4855 for a transitive 1000-node base tournament.

As in most complex systems, in between these two temperature regimes sits an intermediate temperature regime that exhibits the most interesting behaviour. Full spectrum of possible values of Landau's index and expected hitting times can be observed. Indeed, this is the case where transitive structures co-exist with a wide range of hitting times, reflecting different degrees of cycle structures in the strong components. Figure 4 illustrates the relationship between the (inverse) temperature $\beta $ and the hitting times of the dominant 1001-th node. As expected, while extremely high temperatures lead to irreducible base tournaments with intricate cycle structures and long hitting times, very low temperature regimes lead to base tournaments with prominent trivial transitive structures, yielding minimal hitting times. It is also natural to expect that, on average, the expected hitting times will decrease with decreasing temperature, since lower temperatures induce more transitive structures in the base tournaments. The most “complex” and interesting regimes lie in between these two extremes at $\beta \u2248(0.005,0.5)$.

Theorem ^{53} will indicate that the expected hitting times for (1+1)CEA operating on these generated tournaments to be the same as their orders (1001 in this case), which are verified by repeating the experiment with (1+1)CEA. Computations for the expected hitting times are dependent on the probability transition matrix associated with the coevolutionary Markov chain under study. Obviously, different constructions of coevolutionary processes—standard random walk against (1+1)CEA—operating on the same tournaments can lead to different outcomes in the expected hitting time analysis. In contrast, a digraph-theoretic-based approach such as the Landau's index is unaffected by the specific constructions of coevolutionary processes. There are potential ramifications on the construction of sampling-based indices (e.g., hitting times) for analysis of structures in coevolutionary tournaments. Our controlled computational study suggests that more information related to the structures of the tournaments under study may need to be incorporated into those indices so that they are able to uncover those relevant structures. For example, a standard random walk incorporates information of connectivity structures (i.e., out-neighbourhoods of vertices) directly in contrast to the (1+1)CEA.

## 5 Discussion and Conclusion

In this study, we have addressed the issue of studying rich cycle structures associated with coevolutionary systems that use pairwise-comparison-based fitness measures. We have developed a framework that fully captures the underlying structures in coevolutionary problems, and we represent them as cycle structures in digraphs so that we can begin to uncover their effects on coevolution. The digraph representation allows us to give precise characterizations of cycles of different lengths (from 3-cycles to Hamilton cycles that go through every vertex in the digraph) that can occur in coevolutionary problems.

Although these cycle structures appear complex at first given the variety of cycles of different lengths and the manner they can be oriented in digraphs, our study has revealed that coevolutionary digraphs admit a simple qualitative characterization—the corresponding coevolutionary problem is either solvable or not. This has a significant implication on coevolution when it is used as a search procedure. For nonsolvable problems, the presence of Hamilton cycles in the associated digraphs means that coevolution will cycle indefinitely. For each solvable coevolutionary problem, there is a subset of solutions that dominate all other solutions in the solution space. In the context of games, coevolution would be able to search for these dominant strategies in the long run.

The second part of our framework has formalized coevolutionary processes as random walks on digraphs. These random walks are formulated as discrete time processes having the Markov property, which have allowed us to exploit the qualitative characterizations we have made on coevolutionary digraphs to construct relevant quantitative tools for analysis of those processes. We have shown that coevolution of solvable problems corresponds to an absorbing Markov chain with a single absorbing class. For this case, we have shown that the expected hitting time to the absorbing class is the quantity of interest. The other quantity is the limiting invariant distribution of the Markov chain. Although this is of principal interest for coevolution of non-solvable problems, the quantity can also be computed for the case of solvable problems as a result of Lemma ^{50}.

We have demonstrated our quantitative tools through a combination of formal and computational case studies. We have derived expected hitting times and stationary distributions of specific coevolutionary Markov chains operating on transitive and regular tournaments. We have suggested an index for characterizing complexity in coevolutionary problems, and we show how a hierarchy of increasingly complex coevolutionary problems can be generated in a controlled manner that is motivated by statistical physics.

We believe this framework would provide the foundation for existing tools and the means to develop new tools for analysis and design of coevolutionary systems that use pairwise-comparison-based fitness measures. As a case in point, coevolution could *appear to be trapped* in cycles even when it is applied on solvable problems (as observed in various studies such as Chong et al., 2012). Although the associated reducible coevolutionary digraphs can be decomposed into acyclically ordered strong components (by Lemma ^{28}), some of these strong components can be very large digraphs. In the extreme case, a reducible coevolutionary digraph of order $n$ has a strong component of $n-1$ vertices. One possibility is that coevolution would cycle for a long time before *jumping* out and find the single dominant vertex. The other possibility is that the $(n-1)$-sized component is the dominant one, in which case coevolution would seem to cycle indefinitely even though it has found (parts of) the dominant component. In the more realistic case, one could reframe the pairwise-comparison-based fitness measure in the context of *pareto dominance* and make the appropriate change to coevolution to search in the direction of more dominant strong components (Pareto fronts with higher true fitness values in the solution space) (de Jong and Pollack, 2004; de Jong, 2007).

The introduction of this framework for coevolutionary systems provides several directions for future studies. One of them is to provide a more definitive characterization of *problem hardness* affecting coevolutionary systems that include studies on local structures (internal cycle structures within strong components) of coevolutionary digraphs.

Beyond the restricted class of coevolutionary systems we have studied here, the mathematical methodologies we have introduced could be adopted in other formulations of coevolutionary systems. For example, those with population size greater than one and for which coevolutionary fitness measures other than pairwise-comparison (such as the use of average game payoffs) can be investigated. In this case, the entity of interest is the individual state (vertex) representing a finite collection (population) of pure strategies in the context of game-play. The advantage of such an approach is a more informative state space description. In contrast, in evolutionary game theoretic setting studied in the framework of continuous dynamical systems, one usually assumes infinite populations. However, in cases of complex, chaotic coevolutionary dynamics, this can be problematic (Ficici et al., 2005; Tiňo et al., 2013).

Our study is by no means the first one to consider representation of the structure of pairwise relations within the problem to be solved through a digraph. For example, in the multicomponent problems (Bonyadi et al., 2013; Klamroth et al., 2017), the digraph represents pairwise coupling between individual optimization problems that define together the global compound optimization problem to be solved. Typically, one can expect a small number of subproblems (vertices) and there may not exist any coupling between a particular pair of subproblems. In our case, the vertices represent strategies from a potentially huge set of strategies that can be chosen. In addition, the strategy set is fully coupled; that is, between any two different strategies there exists an arc. Most importantly, direct application of Markov chains on such a digraph has the interpretation of a random walk on the strategy space following the outcomes of individual games.

## Acknowledgments

This work was supported by a H2020-MSCA-IF-2014 grant (No. 657027) on “COEVOLFRAMEWORK—Unified Framework for the Analysis of Co-evolutionary Systems.”

## Notes

^{1}

We do consider loops later when we introduce Markov chains, which essentially are weighted digraphs with specific constraints.

^{2}

Cycles of lengths two and one can be defined as well and usually in context of digraphs that admit specific connectivity structures. A 2-cycle is the case whereby two adjacent vertices $u,v\u2208V$ is completely bioriented via $uv$ and $vu$. A 1-cycle is simply the case of a self-loop $uu$.

^{3}

Implicit in our notation are the corresponding sets of arcs in $T(VS(n))$ and $SD(VS(n))$ satisfying constraints for tournaments and semicomplete digraphs, respectively.

^{4}

Standard digraph terminology defines domination in the reverse order, that is, $u$*dominates*$v$ is denoted as $u\u2192v$. Our choice to define domination in this manner becomes clear in the next section when we discuss coevolutionary processes on digraphs and their Markov chain models.

^{5}

We are not describing how one can find the best strategy but how relations of strategies can be captured, for example, *transitive* strategic games represented by *acyclic orientations* of complete graphs.

^{6}

A coevolutionary tournament with one vertex is irreducible since it is a strong tournament by definition. A tournament with two vertices is reducible since there is always one vertex that dominates the remaining vertex in the tournament.

^{7}

Landau's theorem can be generalized further. For example, Reid and Zhang (1998) provided necessary and sufficient conditions for a sequence of integers to be the score sequence of semicomplete digraphs. Moon (1963) showed that the original theorem can be extended to the case whereby one can assign a real number $[0,1]$ to the outcome of the pairwise comparison (game) between a pair of vertices (strategies). That is, for a game between two strategies ($i$ and $j$), $\alpha ij$ is the score for $i$, and $\alpha ji=1-\alpha ij$ is the score for $j$.

^{8}

Loosely speaking, the dynamics of the coevolutionary process as it walks on the digraph is such that it *restarts* in every step. For example, with the (1+1)CEA, the probability that it steps from a particular vertex $xt=u\u2208V(D)$ to an out-neighbour $xt+1=v\u2208ND+(u)$ is independent of the time (generational) step $t$; that is, the probability is the same whether it is $x0=u$ at the start or $xt=u$, which is $t$ generations later.

^{9}

^{10}

^{11}

First hitting and return times are *stopping times* in the sense that the event ${\tau =t}$ depends only on $\Phi 0,\Phi 1,\Phi 2,\u2026,\Phi t$ for $t=0,1,2,\u2026$, which follows from the *strong Markov property* (e.g., Theorem 1.4.2 in Norris (1998)). The Markov property is then said to hold for those random times. Furthermore, given that the coevolutionary Markov chain is discrete time, the strong Markov property always holds (by Proposition 3.4.6 in Meyn and Tweedie, 2009).

^{12}

Given the complete connectivities of coevolutionary digraphs.