A Tutorial on the Spectral Theory of Markov Chains

Markov chains are a versatile tool for modeling stochastic processes and have been applied in a wide variety of scientific disciplines, such as biology, computer science, and finance (Pardoux, 2010). This is unsurprising considering the number of practical advantages they offer: (1) they are easy to describe analytically, (2) in many domains they make complex computations tractable, and (3) they are a well-understood model type, meaning that they offer some level of interpretability when used as a component of an algorithm. Furthermore, as we show in this tutorial, Markov chains are temporal processes that take place on graphs. This makes them particularly suitable for modeling data-generating processes that underlie time series and graph data sets, both of which have received much attention in the fields of machine learning and data mining (Aggarwal, 2015).

The application of Markov chains requires the assumption that at least some aspect of the process being modeled has no memory. An important consequence of this assumption is that the process can be described in detail using a transition matrix. Furthermore, there exists a rich framework for describing distinct features of such processes based on the eigenvalues and eigenvectors of this matrix. This tutorial provides an in-depth exploration of this framework, making use of tools from probability theory, linear algebra, and graph theory. Since the work is intended for readers from diverse academic backgrounds, we concentrate on providing intuition for the tools used rather than strict mathematical formalism.

The material presented underlies multiple methods from different areas of machine learning, and instead of exploring these methods individually, we focus on the general properties that make Markov chains useful across these domains. Nonetheless, so that readers can appreciate the scope of the tutorial, we briefly summarize the methods that it is relevant to. In graph-based unsupervised learning, it is related to nonlinear dimensionality reduction techniques such as Laplacian eigenmaps (Belkin & Niyogi, 2001, 2003) and spectral clustering (Weiss, 1999; Ng et al., 2001; von Luxburg, 2007). These two closely related methods aim to represent data sets in a way that preserves local geometry and are traditionally formulated using graph Laplacians. However, one line of work on spectral clustering instead uses Markov chains (Meilă & Shi, 2000, 2001; Tishby & Slonim, 2001; Saerens et al., 2004; Liu, 2011; Meilă & Pentney, 2007; Huang et al., 2006; Weinan et al., 2008). Furthermore, the method of diffusion maps (Coifman et al., 2005a, 2005b; Coifman & Lafon, 2006) is a generalization of Laplacian eigenmaps that is based on Markov chains and can be tuned to different length scales in a graph, thereby allowing a multiscale geometric analysis of data sets. An in-depth survey of Laplacian eigenmaps, spectral clustering, diffusion maps, and other related methods can be found in Ghojogh et al. (2021). In the domain of time series analysis, the tutorial is relevant to slow feature analysis (SFA) (Wiskott & Sejnowski, 2002), a dimensionality-reduction technique that is based on the notion of temporal coherence and is conceptually related to Laplacian eigenmaps (Sprekeler, 2011). The ideas underlying Laplacian eigenmaps and spectral clustering have also been extended to classification problems, both for labeled (Kamvar et al., 2003) and partially labeled data sets (Szummer & Jaakkola, 2001; Joachims, 2003; Zhou et al., 2005). Finally, the material presented in this tutorial also forms the basis of various approaches to value function approximation in reinforcement learning, such as Mahadevan’s proto-value functions (Mahadevan, 2005; Mahadevan & Maggioni, 2007; Johns & Mahadevan, 2007), Stachenfeld’s work on successor representation (Stachenfeld et al., 2014, 2017), and other closely related methods (Petrik, 2007; Wu et al., 2019). Something common to many of the applications mentioned thus far is that they assume all underlying graphs to be undirected or, equivalently, that the corresponding Markov chain is reversible. This provides a number of guarantees that are crucial for these methods to work, and we explore these guarantees in depth in this tutorial. In most cases, the extension to the directed/nonreversible setting faces a number of challenges and is still actively researched. We discuss these challenges and present various solutions that have been suggested in the literature.

The rest of the text is organized as follows. In section 2, we give a general introduction to discrete-time, stationary Markov chains on finite state spaces and explore some specific types of chains in detail. Section 3 then gives a formal introduction to graphs in order to provide a more detailed description of Markov chains. In section 4, random walks are presented as a canonical transformation that turns any graph into a Markov chain, and the undirected and directed cases are considered separately to better understand the types of Markov chains that they typically give rise to. Finally, in section 5, we explore applications of the material in earlier sections in two areas of computer science literature.

Markov chains can also be depicted visually in the form of a graph, with the state space $S$ drawn as a collection of circles and labeled arrows between these circles representing the nonzero transition probabilities P_ij. We call this diagram the transition graph of a Markov chain. A formal introduction to the mathematics of graphs is given in section 3, but until then, transition graphs are simply used as an illustrative tool.

In Figures 2a to 2d we do this for n = 10 trajectories of length 4, each starting with X₀ = S. Each trajectory is depicted in a specific color, and consists of points in a transition graph plotted across four time points, so that the position of each point indicates a state that one of the trajectories is in at time t. Other than a slight bias toward studying (S) at each time point, it is hard to pick out any clear patterns using only these 10 trajectories. Figures 2e to 2h show similar plots for n = 100, but with all points colored black and the relative occupation of each state for t > 0 indicated by a percentage. Finally, we increase the number of trajectories to n = 1000 in Figures 2i to 2l. Percentages are again used to indicate the relative state occupations, but instead of representing the trajectories with dots, we color each state in gray-scale based on the percentage values. Comparing all the plots in this figure, one can note that in the first row, it is possible to track each of the individual trajectories, whereas in the second and third rows, the focus is instead on approximating the relative probability of doing each activity at each time.

The structure and action of $P$⁠, just like any other matrix, can be evaluated using tools from linear algebra. In particular, $P$ can either multiply column vectors from the left or row vectors from the right. While typical conventions formulate matrix multiplication in the former way, the latter is more common for right stochastic matrices due to its semantic interpretation. Nonetheless, as both operations offer their own insight into the descriptive capacity of Markov chains, we outline both in this section.

A particularly special type of distribution for a Markov chain is one that is invariant under its evolution:

We can also ask how much probability mass moves from one state to another in a given stationary distribution $π$⁠. This is described by the following matrix:

Every real matrix $A∈RN×N$ can be interpreted as a linear transformation. A central task of linear algebra is to shed light on the relationship between the numerical properties of a matrix and various aspects of the transformation that it represents. Often a single matrix represents a combination of several distinct transformations; for example, an object in two or more dimensions can simultaneously be rotated and stretched. Finding the eigenvalues and eigenvectors of a matrix is one way to partition a linear transformation into its component parts and reveal their relative magnitudes. In this section, we give a brief and informal summary of how this works and apply this to transition matrices.

An eigenvector of a matrix $A$ is a vector that is only multiplied by some number λ when multiplied by $A$⁠. Like all other vectors, they can either be rows, that is, $lTA=λlT$⁠, or columns, that is, $Ar=λr$⁠, which are known as left eigenvectors and right eigenvectors, respectively. In both cases, the number λ is called the eigenvalue of the respective eigenvector. It is worth noting that real matrices, including transition matrices, can have complex eigenvalues, $λ∈C$⁠, in which case the corresponding eigenvector is also complex, $v∈CN$⁠. However, such solutions can occur only in complex conjugate pairs, meaning that λ^* and $v*$ are also guaranteed to be an eigenvalue-eigenvector pair, where ^* denotes the complex conjugate.

where m is the number of eigenvalues of $P$ with |λ_ω| = 1. In the long time limit (k → ∞), the terms with |λ_ω| = 1 survive, whereas those with |λ_ω| < 1 die off, with |λ| measuring the rate of decay in the latter case. This allows us to interpret the first and second sums in equation 2.54 to represent the persistent and transient behavior of the Markov chain, respectively.

In the persistent case, we can partition the terms into three types based on the eigenvalues: (i) λ = 1, (ii) λ = −1, and (iii) $λ∈C$⁠, |λ| = 1. We have already seen that a stationary distribution $π$ and the vector $η$ are left and right eigenvectors of type i, and in both cases, these eigenvectors represent fixed structures on the state space that persist when acted on by $P$⁠. To capture this property, we call such eigenvectors persistent structures. In case ii, eigenvectors flip their sign when acted on by the transition matrix, $yTP=-yT$⁠, and return after two steps, $yTP2=yT$⁠. Eigenvectors of this type therefore correspond to permanent oscillations of probability mass between states in $S$⁠, and we therefore refer to them as persistent oscillations. Eigenvalues of type iii are explored in more depth in section 3.5, where we show that they are always complex roots of unity, λ^k = 1 for some k > 2, which occur at equally spaced locations on the unit circle. Therefore, when their corresponding eigenvectors are acted on repeatedly by $P$⁠, they are returned to after k steps, $yTPk=λkyT=yT$⁠. Thus, in analogy to case ii, they describe permanent cycles of probability mass through the state space $S$⁠, and we therefore call such eigenvectors persistent cycles.

In the transient case, an analogous categorization based on the eigenvalues can be applied, consisting of the following three types: (i’) λ ∈ [0, 1), (ii’) λ ∈ (− 1, 0), and (iii’) $λ∈C$⁠, |λ| < 1. For type i’, the corresponding eigenvectors represent perturbations to the persistent behavior that decay over time, $yTP=λyT$ where $|λy|=|λ||y|<|y|$⁠. We therefore call such eigenvectors transient structures. When |λ_j| ≈ 1 these structures describe sets of states that, on average, a chain spends a long time in before converging; these are known as metastable sets (Conrad et al., 2016). Furthermore, λ = 0 can be thought of as a limiting case of type i’ in the sense that any corresponding eigenvector decays infinitely quickly (i.e., $Py=0$⁠) and does not exhibit oscillatory or cyclic behavior. Eigenvalues of type ii’ also decay over time, but like case ii, their negative sign means that the corresponding eigenvectors exhibit oscillatory behavior when acted on by $P$⁠, that is, $yTP=-λyT$ where $|λy|=|λ||y|<|y|$⁠. We refer to eigenvectors of this type as transient oscillations. Eigenvalues of type iii’ generalize those of type iii to the transient case since they are also complex and represent cycles of probability mass. In contrast to type iii, though, they do not occur on the unit circle and need not be equally spaced. Since |λ| < 1, these cycles decay over time, and we therefore call them transient cycles. When |λ| ≈ 1, these cycles can persist for a long time and have been referred to as dominant cycles (Conrad et al., 2016).

The above categorizations are summarized in Table 1, where structures, oscillations, and cycles are colored in green, blue, and red, respectively, and the persistent and transient cases are shaded bright and pale, respectively. In Figure 3, we visualize the six different types of eigenvalue using this color scheme by shading the respective regions of the unit circle in which they can occur.

Before moving on, a couple of details are worth pointing out. First, the above analysis is possible only if the transition matrix $P$ is diagonalizable. One case in which this is guaranteed to hold is for symmetric matrices. However, given that transition probabilities are rarely pairwise symmetric (i.e., P_ij = P_ji), this restriction is clearly too strong. Thus, a more detailed investigation is needed in order to identify the conditions under which the above decomposition can be made, which we do in sections 3 and 4. For a more in-depth account of the theory of diagonalizable matrices, we recommend Meyer (2000). Second, a quick check reveals that all terms in equation 2.54 are dependent on the components w_j that describe the starting distribution (see equation 2.47). Because of this, in the most general case, both the persistent and transient behavior of a Markov chain can be sensitive to initial conditions. In a later section, we consider a particular type of Markov chain for which this is not the case, and provide a simplified analysis of their evolution over time.

In the following sections, we explore three types of Markov chains. In order to describe each type in detail, we first define various properties that apply to individual states, or sets thereof, in a state space $S$⁠. This is the focus of this section.

We start by making the following definitions related to how states in $S$ are connected:

Communication is a useful property for describing states in a Markov chain, as exemplified by the following result:

Furthermore, we can make a useful categorization of Markov chains based on the number of communicating classes they have:

In words, an irreducible Markov chain is one in which for any pair of states, there exists a connecting path in both directions. In Figure 4, three example Markov chains are shown, with the communicating classes indicated by the dashed boxes. Take a moment to double-check why each state belongs to its communicating class, and verify that the examples in Figures 4a and 4b are reducible, whereas the one in Figure 4c is irreducible. Finally, observe from Figure 4b that it is possible for states in one communicating class to be accessible from states in another class (e.g., s₃ → s₄).

Irreducible Markov chains feature in a number of subsequent sections of this tutorial due to the following result:

This result can be applied to the example of Figure 4c by using the observation of the previous section that stationary distributions of a given Markov chain are eigenvectors of the associated transition matrix with eigenvalue 1. If we were to find the transition matrix of the chain in Figure 4c and compute its eigenvectors, we would indeed find a single left eigenspace of $P$ corresponding to eigenvalue 1, with strictly positive elements. When normalized such that the row sum is 1, the resulting vector is $π=(π1,π2,π3,π4,π5,π6)T=(0.09,0.09,0.07,0.31,0.18,0.26)T$⁠, which we encourage readers to check themselves. For reducible Markov chains, the guarantees of uniqueness and positivity of the stationary probabilities no longer hold. In order to explore the stationary distributions of such chains, we introduce some new concepts for describing states.

Each state $si∈S$ can be categorized based on how likely it is to be revisited, given that it is currently occupied. This is formalized by the following definition:

This is a useful way to characterize states in $S$⁠, since it generalizes to all states within a communicating class, that is, for any communicating class $C⊆S$⁠: either all states in C are recurrent or all states in C are transient. Transience and recurrence are therefore examples of class properties, and we henceforth use the terms recurrent class and transient class for communicating classes that contain recurrent and transient states, respectively. Furthermore, for finite state spaces, a Markov chain is guaranteed to have at least one recurrent class. Building on this, we can then apply the notion of recurrence to a Markov chain as a whole:

As an illustration, we apply these concepts to the reducible chains depicted in Figure 4. In Figure 4a, there are two communicating classes, and in each one, there is no possibility to exit. This means that if a state in one of these classes is occupied, it is guaranteed to be revisited at some future time step, that is, f_i = 1 for all states in each class. Therefore, both classes are recurrent and the chain as a whole is recurrent. In Figure 4b, the main difference is that there is now the possibility to exit the blue class without returning. For example, assuming s₁ is occupied, although there is a possibility that s₁ can be visited again later (e.g., s₁ → s₂ → s₁), as soon as a transition s₁ → s₄ takes place, this will no longer be possible. This is why f_i < 1 for states in the blue class. Hence, while the red class is recurrent, the blue class is transient, and because of this, the chain as a whole is not recurrent.

For an irreducible Markov chain, such as the example shown in Figure 4c, every state is guaranteed to be recurrent, leading to the following proposition:

However, from the example in Figure 4a, it is clear that the converse does not hold, since it is also possible for a reducible chain to be recurrent. In fact, whether a reducible chain is recurrent or not determines certain features of the stationary distributions belonging to the chain. This is outlined by the following proposition:

We can apply these results to the two reducible chains in Figure 4. In the example of Figure 4a, the stationary distribution associated with the blue class is $π1=(0.35,0.36,0.29,0,0,0)T$ and the one associated to the red class is $π2=(0,0,0,0.39,0.26,0.35)T$⁠. We can then generate an arbitrary number of extra stationary distributions by taking convex combinations of $π1$ and $π2$ with coefficients α₁ and α₂. For example, with α₁ = 0.25 and α₂ = 0.75, we get $π=(0.09,0.09,0.07,0.29,0.2,0.26)T$⁠. Furthermore, the last bullet point in proposition 3 tells us that since this chain is recurrent, we can be sure that any convex combination with positive coefficients yields a distribution with strictly positive entries. This property of recurrent chains is particularly relevant to our treatment of both reversible chains in section 2.6 and random walks in section 4, and we henceforth use $π>0$ to denote a stationary distribution with this property. In the example of Figure 4b, the red class is the only recurrent class, meaning that the stationary distribution associated with this class is the only stationary distribution of the chain. Since the transition probabilities for states in this class are the same as in the example of Figure 4a, this stationary distribution is $π=π2$⁠. Furthermore, in agreement with proposition 3 we see that the transient states in this chain have a stationary probability of 0.

The notion that states can be revisited is also meaningful in another sense. We define the following quantity, which describes how frequently such revisits can take place.

Like transience and recurrence, period is also a class property, and we use d to refer to the period of a whole class. This in turn allows us to define the period of a Markov chain:

Consider the example shown in Figure 5a. This chain has two communicating classes, the red one being a transient class with d = 2 and the blue one being a recurrent class with d = 1, meaning that this chain has mixed periodicity. In Figures 5b–5d we show three irreducible Markov chains, with panels b and c having period 3 and 2, respectively, and panel d being aperiodic.

Markov processes are a broad class of models, and even under the restricted settings considered in this tutorial (discrete time, homogeneous, and finite state spaces), there are many distinct types of chains. In the following sections, we concentrate on three particular types that are relevant in applied domains.

When modeling a system that evolves over time, it is important to ask what can be said, if anything, about its long-term behavior. For a Markov chain, this question can be phrased in two ways. On one hand, we can sample a single trajectory starting from some initial state and ask what the average behavior is over time: How often is it found in each state $si∈S$ for a trajectory of length t? On the other hand, we can describe our starting conditions as a distribution $μ(0)$ and ask what this evolves to in the future: What is the probability of being in each state s_i at a later time t? We refer to these two notions of long-term behavior as the trajectory and distribution perspectives, respectively. While the analyses given so far predominantly use the latter perspective, we remind readers that in section 2.1, we introduced the idea of a distribution over $S$ by taking the limit of an infinitely large ensemble of trajectories, meaning that the two concepts are closely related.

This two-way view originates from the field of statistical physics, where physical processes can either be analyzed with temporal averages (i.e., the trajectory perspective) or ensemble averages (i.e., the distribution perspective). One class of systems that has received a lot of study in this field are those for which these two types of averaging yield the same result as t → ∞. Such systems are known as ergodic systems, and this equivalence means that a statistical description of their long-term behavior can be described simply by a single, sufficiently long sample. One implication of this is that initial conditions are forgotten over time, which makes ergodic systems particularly attractive from a simulation or modeling perspective. Finally, with this in mind, we can define an ergodic Markov chain as follows:

Clearly, in order for a chain to be ergodic, it must have a unique stationary distribution. Therefore, by virtue of theorem 1, a necessary condition for a chain to be ergodic is that it is irreducible. However, there is no guarantee that an irreducible chain converges, which is the second condition of definition 12. The convergence of a Markov chain is related to its periodicity, as explained by the following result:

We can apply this result to the irreducible Markov chain in Figure 5c. This chain has a unique stationary distribution $π=(0.35,0.15,0.5)T$ and a period of d = 2. Therefore, there is no guarantee that the chain converges to $π$ since it can get trapped in persistent oscillations. To observe this, we can try out different initial conditions and iteratively apply the update rule in equation 2.12. For example, starting with the distribution $μ(0)=(0.25,0.5,0.25)T$⁠, we get a persistent oscillation between the following two distributions: $μ(k)=(0.175,0.075,0.75)T$ and $μ(k+1)=(0.525,0.225,0.25)T$⁠. However, this is not the only persistent oscillation possible for this chain, which can be observed by trying out different initial conditions. Finally, in section 3.5, we gain more insight on theorem 2 by using tools from graph theory to describe the eigenvectors of $P$ with |λ| = 1.

A key insight from theorem 2 is that a Markov chain is guaranteed to converge only if it is aperiodic. Together with irreducibility, this therefore provides the conditions under which a chain is ergodic:

It is worth emphasizing that since no assumption is made in definition 13 about the number of communicating classes, it also applies in the case of reducible recurrent chains where there is an infinite number of distributions $π>0$ to choose from. However, in such cases, the choice of $π$ makes no difference:

Moreover, the set of stationary distributions $π>0$ belonging to a recurrent Markov chain is the same as the set belonging to the corresponding time reversal:

A number of observations can be made about this definition. First, the left (right) terms represent the flow of probability from s_i to s_j (s_j to s_i), given that the chain is described by distribution $π$⁠. Thus, for a reversible Markov chain in one of its stationary distributions, the flow from one state to another is completely balanced by the flow in the reverse direction, meaning that the flow matrix $Fπ$ is always symmetric for such chains. By comparison with equation 2.19, we see that detailed balance is a stronger condition than global balance, since in the latter case, there is only an equivalence between the total flow in and out of each state. Second, since π_i and π_j are nonzero, it follows that P_ij ≠ 0 if and only if P_ji ≠ 0. Thus, the transition structure of a reversible Markov chain always permits the return to the previous state, and because of this, the period of a reversible Markov chain can be at most 2. Third, while some sources assume irreducibility as a precondition of reversibility, we instead base our definition on the weaker condition of recurrence (Porod, 2021). This is due to the fact that we only need recurrence in order to define the time reversal of a Markov chain. Furthermore, with this convention, theorem 4 applies more broadly to reducible Markov chains, which lets us make a closer comparison between reversible Markov chains and undirected graphs in section 4. Finally, theorem 4 implies that there are two distinct ways in which Markov chains can be nonreversible: (1) they can be recurrent without satisfying detailed balance, or (2) they can be nonrecurrent. In case 1, $ΠP≠PTΠ$ for any distribution $π>0$⁠, meaning that the flow matrix $Fπ$ is asymmetric, and in case 2, no positive stationary distribution exists. Finally, note that for a nonrecurrent chain, there exists the possibility that $Fπ$ is symmetric for all stationary distributions despite none of those distributions being strictly positive. For such chains, removing all transient states from $S$ produces a reversible chain. We therefore refer to such chains as semireversible.

To illustrate some of these points, we consider four Markov chains in Figure 6, with panels a, d, g, and j showing the transition graphs of each example, and panels b, e, h, and k showing the respective transition matrix, stationary distribution, and flow matrix (for simplicity, each example has a single recurrent class, so that both the stationary distribution and the associated flow matrix are unique). As a visual illustration of the pairwise stationary flow between states in each example, in panels c, f, i, and l, the stationary distribution is represented as a bar plot, with the portions of probability mass flowing in (left) and out (right) of each state shown as portions of each bar. The example in panel a is reversible, as can be seen by the symmetry of $F1π$ in panel b, or equivalently by the matching between left and right portions of all bars in panel c. The example in panel d is almost equivalent to the one in panel a, except that the outgoing transition probabilities from s₁ have been slightly modified (indicated by the colored arrows in panels a and d and the colored entries of $P1$ and $P2$ in panels b and e). This modification is enough to violate detailed balance, as can be seen by the asymmetry of $F2π$ or the bar plot in panel f. Finally, the chains depicted in panels g and j are nonrecurrent since in both cases, state s₃ has only outgoing transitions. Therefore, π₃ = 0, and both examples are nonreversible. However, a quick check of $F3π$ or the bar plot in panel i reveals that the example in panel g is semireversible. Conversely, the example in panel h is identical to the one in panel g except for the outgoing transitions from s₄ (again indicated by the colored arrows in panels g and j and the colored entries of $P3$ and $P4$ in panels h and k), which leads to an asymmetric stationary flow between the recurrent states.

It is worth pointing out that in our analysis above, we check for reversibility by inspecting the stationary distributions and the corresponding flow matrices of each example. However, since reversibility is a property associated with Markov chains and not with distributions, one might wonder whether there is an alternative way to formalize it based purely on the transition probabilities P_ij. Clearly, equation 2.71 prohibits one-way transitions (i.e., P_ij > 0 and P_ji = 0), but this is only a necessary condition of reversibility. Can we offer anything more precise? Fortunately, the answer is yes, and it is given by Kolmogorov’s criterion (Kolmogoroff, 1936):

One way to understand this theorem is that for reversible Markov chains, the probability of traversing any closed path in the state space $S$ is independent of the direction of traversal. Hence, reversible Markov chains can be thought of as having zero net circulation. By contrast, recurrent Markov chains that are nonreversible have at least one path that violates equation 2.72, over which there is a higher probability to traverse in one direction than the other. For the example in Figure 6a, the relevant closed paths are (up to a cyclic permutation): (1) s₁ ↔ s₂ ↔ s₄ ↔ s₃ ↔ s₁, (2) s₁ ↔ s₂ ↔ s₄ ↔ s₁, and (3) s₁ ↔ s₄ ↔ s₃ ↔ s₁. In any of these cases, going around clockwise is equally probable as going around counterclockwise, which is to be expected since this Markov chain is reversible. The example in Figure 6d has the same closed paths available, except that the outgoing transition probabilities from s₁ have been changed. This small adjustment is enough to introduce circulation on all the closed paths: for both paths 1 and 3 the counterclockwise direction is more probable since P₁₃P₃₄P₄₂P₂₁ > P₁₂P₂₄P₄₃P₃₁ and P₁₃P₃₄P₄₁ > P₁₄P₄₃P₃₁, respectively, and for path 2, the clockwise direction is more probable since P₁₂P₂₄P₄₁ > P₁₄P₄₂P₂₁. Therefore, by virtue of having at least one path with net circulation, equation 2.72 confirms that this chain is indeed nonreversible.

These analyses illustrate how theorems 4 and 5 provide two alternative but equivalent definitions of reversibility. Something common to both of these interpretations is that reversible Markov chains satisfy a type of equilibrium, either between the exchange of probability mass between pairs of states or the circulation along closed paths, respectively. In fact, the concept of detailed balance stems from early work in the field of statistical mechanics aimed at formalizing the notion of thermodynamic equilibrium on a microscopic level (Gorban, 2014). More recently, Markov chain Monte Carlo methods, which are predominantly based on reversible ergodic chains (see section 5.1), have received widespread application in the natural sciences as a way to model systems that are in thermodynamic equilibrium (Richey, 2010). Conversely, Markov chains that violate detailed balance, or equivalently those with net circulation, have been applied to the less-well-understood case of systems which are out of equilibrium (Jiang et al., 2004; Zhang et al., 2012; Ge et al., 2012). Furthermore, their stationary distributions have been referred to as nonequilibrium steady states (NESS) (Jiang et al., 2004; Zhang et al., 2012; Ge et al., 2012; Conrad et al., 2016; Witzig et al., 2018), which reflects the fact that such distributions are kept fixed over time via unequal flows of probability mass between states (⁠$π2$ is an example of a NESS, as can be seen in the bar plot in Figure 6f).

Reversible Markov chains are significantly easier to treat both analytically and numerically than nonreversible chains. Because of this, there exist various procedures for modifying a nonreversible Markov chain so that it becomes reversible, which is sometimes referred to as reversibilization (Fill, 1991; Brémaud, 1999). For a recurrent chain, this can be done by taking an average of the forward and backward transition probabilities, $P$ and $Prev$⁠, that describe the chain and its time reversal, respectively. This averaging process can be either additive or multiplicative, leading to the following two definitions:

Finally, one concept in the theory of Markov chains that is particularly relevant to applied domains is absorption. A state $si∈S$ is called absorbing if it is possible to transition into the state but not out of it, meaning that P_ii = 1 and the chain stays in s_i for all future time steps. An absorbing Markov chain is one for which from every state $si∈S$⁠, there exists some path to an absorbing state. Since it is possible to start in a nonabsorbing state and never return, all nonabsorbing states are transient, and the presence of such states means that absorbing chains can be neither reversible nor ergodic. Absorbing chains often occur in Markov decision processses (MDPs), which are central to the field of reinforcement learning (Sutton & Barto, 2018).

The possible transitions in an absorbing chain can be partitioned into three types: (1) transient → transient, (2) transient → absorbing, and (3) absorbing → absorbing. Although the assignment of indices to states in $S$ is arbitrary, an assignment based on this partitioning simplifies the analysis of absorbing Markov chains.

We depict this partitioning of transition probabilities in Figure 7a. An absorbing chain with one absorbing state is shown, with the transitions belonging to $Q$⁠, $R$⁠, and $1$ colored black, red, and blue, respectively. Furthermore, in Figure 7b, we show the matrices $Q$ and $R$⁠.

Since any transient state can reach an absorbing state in a finite number of steps, the probability that the chain ends up in an absorbing state at some future time is 1. For this reason, in the infinite time limit, we can expect to see no transitions taking place between transient states, that is, $limn→∞Qn=0$⁠. This is an advantageous property, since it means that if we sum up all powers of $Q$⁠, known as the Neumann series of $Q$⁠, then the contributions for larger powers get progressively smaller and the sum converges to $(1-Q)-1$ (see Meyer, 2000, p. 618). Calculating this sum for $Q$ leads to the following useful quantity, which relates transient states in $S$ (Porod, 2021):

When analyzing an absorbing chain, it is very handy to have access to the fundamental matrix. By taking into account all nonnegative powers of $Q$⁠, it contains information about all possible paths available between pairs of transient states. Because of this, it is a useful predictive tool that allows several properties of the Markov chain to be deduced (Porod, 2021). Furthermore, in the field of reinforcement learning, it is closely related to the successor representation (Dayan, 1993) (see section 5.2).

This concludes our exploration of different types of Markov chains. In Figure 8, we provide a summary of the material presented in this section in the form of a Venn diagram. In this diagram, each type of Markov chain is drawn as a circle or ellipse, with defining properties and results listed in each case. Take a moment to look at this image, and pay attention to the overlapping regions, which indicate how different types of chains are related. For a more in-depth presentation of the material in this section, we recommend Porod (2021) and Brémaud (1999).

In the next section, we introduce graphs as an alternative way to describe Markov chains and summarize insights that emerge from this description. Then, in section 4, we explore the connection between graphs and Markov chains in more depth using the notion of random walks, which allows various relationships to be made between specific types of graphs and some types of Markov chains introduced in this section.

So far, we have implicitly been interpreting Markov chains as graphs whenever we draw a transition graph. In this section, we formally introduce the concept of graphs, which provides a foundation to the material on random walks in section 4. Readers should note that definitions in graph theory often vary among different sources. Here we use a convention that can encompass a wider variety of graphs, thereby offering greater generality.

A graph G = (V, E) is a set of N vertices V = {v₁, v₂, . . . , v_N} together with an edge set E containing pairs of vertices in V. Conceptually, V might represent a collection of objects and E a specification of how some pairs in this collection are related to one another. A natural way to categorize graphs is based on the way in which edges are defined. For instance, in an undirected graph, each edge has no direction and is typically denoted as (v_i, v_j) ∈ E, whereas in a directed graph, each edge has a specified starting and ending vertex and is usually denoted as (v_i → v_j) ∈ E. Examples of undirected and directed graphs can be seen in the first and second rows of Figure 9, respectively. Unless otherwise stated, we depict undirected edges as straight lines and directed edges as curved lines with arrowheads indicating the direction. A second distinction we can make is between unweighted graphs, in which one only cares about whether two vertices are related or not, and weighted graphs, in which each edge has a positive weight w_ij describing the strength of the relationship.³ In Figure 9, the examples in the first column are unweighted and all other graphs are weighted, with weights indicated by numbers next to each edge. The type of edges that a graph has is often chosen based on the type of relationship that one wants to describe. For example, assume that we have a graph G where vertices represent PhD students. Then, if we want to represent the relationship of being in the same research group, undirected, unweighted edges are a natural choice (such as Figure 9a). Conversely, if we want edges to describe whether one student has participated on a main project of another student, then this clearly requires directed unweighted edges (such as Figure 9d). If we now consider variants of the first and second examples, instead focusing on how similar the research topics of two students are or how much work one student has contributed to another student’s project, then we now need undirected weighted and directed weighted edges, respectively (such as Figures 9b and 9c and Figures 9e and 9f). It is worth noting that in order to assign weights to edges, one needs to specify a scale on which to measure the strength of relationship between vertices.

One can also describe graphs based on their connectivity. In an undirected graph, if there exists a path between each pair of vertices, then the graph is said to be connected; otherwise it is disconnected. The notion of connectivity can generalize to disconnected graphs if we instead consider subsets of vertices in G, which are known as subgraphs. Any subgraph that is connected but is not part of any larger connected subgraph is called a connected component. Both of the undirected graphs in Figures 9a and 9b are connected, whereas Figure 9c shows an example that is disconnected, with two connected components. In particular, this latter example even has a vertex that does not have any edges at all; it is known as an isolated vertex. For a directed graph, if there are directed paths running from v_i to v_j and from v_j to v_i for all pairs of vertices v_i, v_j ∈ V, then the graph is said to be strongly connected. Alternatively, a directed graph is weakly connected if for all pairs of vertices v_i, v_j ∈ V, it is possible to get from v_i to v_j and from v_j to v_i by any path, ignoring the direction of the edges. Clearly, a directed graph is weakly connected if it is strongly connected, but not vice versa. Furthermore, strongly or weakly connected subgraphs that are not part of any larger such subgraphs are referred to as strongly or weakly connected components, respectively. The directed graphs in Figures 9d and 9e are strongly connected, whereas the one in Figure 9f is only weakly connected and has two strongly connected components (take a moment to verify this).

For the remainder of this tutorial, we assume that the graphs we deal with are both weighted and directed. The reason we choose this convention is that it is more general. On the one hand, any unweighted graph can be considered as a special case of a weighted graph where the weights are all set to 1. Thus, we henceforth only talk about weight matrices as opposed to adjacency matrices when describing graphs numerically. On the other hand, there is one sense in which directed graphs can be thought of as a generalization of undirected graphs. If v_i and v_j are two distinct vertices that share an edge in an undirected graph, then the weight of this edge is guaranteed to appear twice in the weight matrix $W$ by virtue of being symmetric. If instead these vertices belong to a directed graph and there is an edge (v_i → v_j), then this edge appears only once in $W$⁠. Therefore, it is possible to interpret an undirected edge between v_i and v_j as being equivalent to a pair of directed edges of the same weight, with one connecting v_i to v_j and the other connecting v_j to v_i. This is the interpretation we use throughout the rest of the tutorial whenever we refer to undirected graphs. As an example, in Figure 10, we show two equivalent depictions of an undirected graph, with the top image drawn in the usual way and the bottom image drawn using pairs of directed edges. Below these two drawings, the weight matrix of this graph is shown. One must note that interpreting undirected graphs in this way is somewhat atypical; however, it allows us a greater level of generality when dealing with different types of graphs in section 4. Furthermore, this interpretation only applies to edges between distinct vertices; edges that connect vertices to themselves are discussed in section 3.4.

Once the weight matrix of a graph is known, it is easy to calculate the total weight coming in and out of each vertex. The total incoming weight of a vertex v_i can be found by summing over the ith column of $W$ and is known as the in-degree of v_i: $di-:=∑j=1NWji$⁠. Conversely, the total outgoing weight of v_i is calculated by the sum over the ith row of $W$⁠, $di+:=∑j=1NWij$⁠, and is known as the out-degree of v_i. Since undirected graphs always have bidirectional edges and symmetric weight matrices, the in- and out-degrees of such graphs are always equal and are simply referred to as vertex degrees, denoted by d_i.⁴ For example, in the graph of Figure 10 the degrees are d₁ = 3, d₂ = 4, and d₃ = 1. In the more general case of directed graphs, there is no guarantee that $di+=di-$⁠. However, summing over all in- or out- degrees for any graph always produces the same number, $vol(G)=∑i=1Ndi+=∑i=1Ndi-=∑i=1N∑j=1NWij$⁠, which is sometimes referred to as the volume of G. As an example, consider the directed graph in Figure 9e: each vertex has different in- and out-degrees, i.e. $d1-=1$⁠, $d1+=0.5$⁠, $d2-=2.5$⁠, $d2+=3$⁠, $d3-=0.8$⁠, $d3+=1$⁠, $d4-=2$⁠, and $d4+=1.8$⁠, but summing over either of the degree types yields vol(G) = 6.3. Nonetheless, some directed graphs can have $di+=di-$ for each vertex, and such cases are known as balanced graphs (Banderier & Dobrow, 2000; Aldous & Fill, 2002). In keeping with the notation of undirected graphs, we denote the vertex degrees of a balanced graph as $di=di+=di-$⁠. An example of a balanced graph along with its corresponding weight matrix is shown in Figure 11, and a quick check reveals that summing over the rows or columns of $W$ indeed yields the same values. Just as a balanced graph is a special case of a directed graph, we can similarly say that an undirected graph is a special case of a balanced graph, and this interpretation is important in section 4.

In the examples considered so far, all edges connect pairs of vertices that are distinct, that is, v_i ≠ v_j. While this is sometimes enforced as a rule, some conventions also allow edges to connect vertices to themselves, which are known as self-loops. For undirected graphs, the standard convention is that a self-loop at vertex v_i counts doubly to the vertex degree d_i, while other edges only count singly, that is, d_i = 2 × W_ii + ∑_{j ≠ i}W_ij. This somewhat counterintuitive property is typically demonstrated using the degree sum formula (West, 2001). For undirected graphs, this states that each edge contributes twice its weight to the volume. Since self-loops only involve a single vertex, the only way that this rule can be respected is if they count twice as much to the vertex degrees as other edges. A property that we require when dealing with undirected graphs in section 4 is that the vertex degrees are calculated by the row sums of $W$⁠. Clearly, this property is violated by the factor of 2 that applies to undirected self-loops. As a result, in this tutorial we assume that self-loops are always directed, regardless of whether they occur in undirected or directed graphs. This is an atypical definition, since undirected graphs typically are not allowed to have directed edges. However, as can be seen from the examples in Figure 12, this preserves the fact that undirected graphs have symmetric weight matrices, whereas directed graphs have nonsymmetric weight matrices, which is sufficient for the scope of this tutorial.

We close this section by noting some similarities between our definitions of Markov chains and graphs. First, the transition matrices of Markov chains, like the weight matrices of graphs, are nonnegative. Second, in a directed graph, any entry W_ij ≠ 0 of the weight matrix describes an outgoing edge from vertex v_i to v_j, and analogously, any entry P_ij ≠ 0 of a transition matrix describes an outgoing transition probability from s_i to s_j. Putting these together, we see that in the most general sense, any Markov chain can be thought of as a directed graph, with $P$ being the associated weight matrix. Indeed, this interpretation is precisely what justifies us in visualizing a Markov chain by its transition graph. In the next section, we present some useful results that emerge as a result of this way of thinking about a Markov chain. Finally, for a comprehensive text on graph theory that covers much of the material in this section, we recommend West (2001).

Nonnegative matrices have received widespread attention in mathematics, and in particular their eigenvalues and eigenvectors are the focus of spectral graph theory (Chung, 1997). In this section, we apply some results from this field to transition matrices, considering first irreducible chains and subsequently exploring the generalization to reducible chains.

A fundamental result used in spectral graph theory is the Perron-Frobenius theorem, and while a full treatment of it is beyond the scope of this tutorial, we now summarize its key implications for transition matrices of irreducible Markov chains.

To illustrate the above theorem, in Figures 13a to 13c, we show the transition graphs and eigenvalue plots of three irreducible Markov chains. The first observation to make is that, in agreement with theorem 6, λ = 1 is an eigenvalue in each case and occurs only once. Furthermore, as a quick exercise, we encourage readers to find the eigenvectors of λ = 1 for each example and normalize them to obtain $π$ and $η$⁠. Finally, the eigenvalue plots show that in each example, all eigenvalues indeed lie either on the unit circle (|λ| = 1) or within it (|λ| < 1).

Using our terminology from section 2.3, eigenvalues within the unit circle represent transient structures, transient oscillations, and transient cycles. Of the irreducible chains in Figure 13, only panels a and b have eigenvalues of this type, and in both cases, they are complex conjugate pairs describing transient cycles. Looking at the transition probabilities in each example, it is clear that these transient cycles flow clockwise around the state space.

On the other hand, eigenvalues on the unit circle other than $λ=1$ represent persistent oscillations and persistent cycles. Theorem 2 tells us that these are possible only when a chain is periodic. The following result sheds light on this by relating the eigenvalues with |λ| = 1 to the period of a chain (Gebali, 2008):

In simple terms, proposition 6 says that the eigenvalues of $P$ with modulus 1 are always dth roots of unity. We can verify this by checking the periodic examples in Figures 13b and 13c. In both cases, the number of eigenvalues on the unit circle is indeed equal to the period of the chain and they are also equally spaced. Furthermore, proposition 6 offers an alternative perspective on how the periodicity affects the persistent behavior of a Markov chain. For example, the chain in Figure 13a has only a single eigenvalue on the unit circle, corresponding to its unique stationary distribution. It is therefore guaranteed to end up in this distribution since all other eigenvalues have |λ| < 1. This is equivalent to the statement that this chain is ergodic, which a quick check of the transition graph confirms. Conversely, the chain in Figure 13b has an additional eigenvalue λ = e^πi = −1 on the unit circle by virtue of the fact that it has period 2. Therefore, its persistent behavior can only be fully described using both the unique stationary distribution $π$ and the eigenvector $y$ associated with λ = −1. For example, we know from theorem 2 that such a chain can get trapped in a persistent oscillation (i.e., $μ1→μ2→μ1→μ2→...$⁠). For any such oscillation, $μ1$ and $μ2$ can always be expressed as a linear combination of $π$ and $y$⁠, meaning that this sequence indeed oscillates between two points in the space spanned by these eigenvectors. While this example only involves real eigenvalues and therefore only real eigenvectors, the interpretation extends to d > 2, for which proposition 6 tells us that there must be complex eigenvalues with |λ| = 1. For example, the chain in Figure 13c has period d = 3, and it has the following three eigenvalues on the unit circle: λ₁ = 1, $λ2=e2πi3$⁠, and $λ3=e4πi3$⁠. Analogous to the d = 2 case, any persistent cycle of this chain can be expressed using the three corresponding eigenvectors, which in the case of λ₂ and λ₃ must have complex entries. Rather interestingly, this means that for chains with period d > 2, persistent cycles are cycles in a complex space despite being sequences of real valued distributions.

Applying spectral graph theory to the transition matrices of reducible Markov chains produces a weaker set of results. For example, the generalization of theorem 6 to the reducible chains is the following:

To understand theorem 7 in more depth, consider the example in Figure 13d. This Markov chain has two recurrent communicating classes, which means that λ = 1 has a multiplicity of 2. We indicate this on the eigenvalue plot by a larger size circle. Furthermore, we color this circle half red and half blue to reflect the fact that we can choose the eigenvectors for λ = 1 based on the two recurrent communicating classes, for example, $π1=(0.58,0.42,0,0)T$ and $η1=(1,1,0,0)T$ as a pair of left and right eigenvectors for the red class and $π2=(0,0,0.43,0.57)T$ and $η2=(0,0,1,1)T$ as a pair of left and right eigenvectors for the blue class. Then, $π1$ and $π2$ together span the left eigenspaces of λ = 1 (including all possible stationary distributions), whereas $η1$ and $η2$ span the right eigenspaces of λ = 1. It is worth emphasizing that while there is an infinite number of other ways to choose the basis vectors for λ = 1, this is a convenient choice since it is the only one for which all basis vectors have strictly nonnegative entries. For example, consider the choice of $π˜1=13π1+23π2=(0.19,0.14,0.29,0.38)T$ and $η˜1=η=(1,1,1,1)T$ as a first pair of eigenvectors. If we then choose a second pair $π˜2$ and $η˜2$ that satisfy biorthogonality (see equation 2.46) within this space, they are guaranteed to contain negative entries, for example, $π˜2=π1-π2=(0.58,0.42,-0.43,-0.57)T$ and $η˜2=3η1-32η2=(3,3,-32,-32)T$⁠. Thus, the choice of eigenvectors stated in theorem 7 is in some sense special since it is the only one that preserves our intuition that left eigenvectors with λ = 1 correspond to distributions over the state space $S$⁠. For this reason, we henceforth assume this convention when referring to eigenvectors with λ = 1.