The graphical brain: Belief propagation and active inference

This paper attempts to describe functional integration in the brain in terms of neuronal computations. We start by asking what the brain does, to see how far the implicit constraints on neuronal message passing can take us. In particular, we assume that the brain engages in some form of (Bayesian) inference—and can therefore be described as maximizing Bayesian model evidence (Clark, 2013; Friston, Kilner, & Harrison, 2006; Hohwy, 2016; Mumford, 1992). This implies that the brain embodies a generative model, for which it tries to gather the greatest evidence. On this view, to understand functional integration is to understand the form of the generative model and how it is used to make inferences about sensory data that are sampled actively from the world. Happily, there is an enormous amount known about the various schemes that can implement this form of (Bayesian) inference, thereby offering the possibility of developing a process theory (i.e., neuronally plausible scheme) that implements the normative principle of self-evidencing (Hohwy, 2016).

In brief, this (rather long) paper tries to integrate three themes to provide a rounded perspective on message passing or belief propagation in the brain. These themes include (a) the formal basis of belief propagation, from the perspective of the Bayesian brain and active inference; (b) the biological substrates of the implicit message passing; and (c) how discrete representations (e.g., semantics) might talk to representations of continuous quantities (e.g., visual contrast luminance). Technically, the key contributions are twofold: first, the derivation of belief propagation and Bayesian filtering in generalized coordinates of motion, under the framework afforded by factor graphs. This derivation highlights the similarities between representations of trajectories over future time points, in discrete models, and the representation of trajectories in generalized coordinates of motion in continuous models. Second, we described a fairly generic way in which discrete and continuous representations can be linked through Bayesian model selection and averaging. To leverage these technical developments, for an understanding of brain function, we highlight the constraints they offer on the structure and dynamics of neuronal message passing, using coarse-grained evidence from anatomy and neurophysiology. Finally, the nature of this message passing is illustrated using simulations of pictographic reading.

In what follows, we use graphical representations to characterize message passing under deep (hierarchical) generative models that might be used by the brain. We use three sorts of graphs to emphasize the form of generative models, the nature of Bayesian belief updating, and how this might be accomplished in neuronal circuits—both at the level of macroscopic cortical hierarchies and at the more detailed level of canonical microcircuits. The three sorts of graphs include Bayesian networks or dependency graphs (MacKay, 1995; Pearl, 1988), where nodes correspond to unknown variables that have to be inferred and the edges denote dependencies among these (random) variables. This provides a concise description of how (e.g., sensory) data are generated. To highlight the requisite message passing and computational architecture, we will use Forney or normal style factor graphs (Loeliger, 2002). In Forney factor graphs, the nodes now represent local functions or factors of a probability distribution over the random variables, while edges come to represent the variables per se (or more exactly a probability distribution over those variables). Finally, we will use neural networks or circuits where the nodes are constituted by the sufficient statistics of unknown variables and other auxiliary variables, such as prediction errors. The edges in these graphs denote an exchange of (functions of) sufficient statistics—of the sort one can interpret in terms of neuronal (axonal) connections. Crucially, these graphical representations are formally equivalent in the sense that any Bayesian network can be expressed as a factor graph. And any message passing on a factor graph can be depicted as a neural network. However, as we will see later, the various graphical formulations offer different perspectives on belief updating or propagation. We will leverage these perspectives to work from a purely normative theory (based upon the maximization of Bayesian model evidence or minimization of variational free energy) towards a process theory (based upon belief propagation and the attractors of dynamical systems).

In this paper, we use Forney factor graphs for purely didactic purposes; namely, to illustrate the simplicity with which messages are composed in belief propagation—and to emphasize the recurrent aspect of message passing. However, articulating a generative model as a Forney factor graph has many practical advantages, especially in a computer science or implementational setting. Factor graphs are an important type of probabilistic graphical model because they facilitate the derivation of (approximate) Bayesian inference algorithms. When a generative model is specified as a factor graph, latent variables can often be inferred by executing a message passing schedule that can be derived automatically. Examples include the sum-product algorithm (belief propagation) for exact inference, and variational message passing and expectation propagation (EP) for approximate inference (Dauwels, 2007). Probabilistic (hybrid or mixed) models (Buss, 2003) that include both continuous and discrete variables require a link factor, such as the logistic or probit link function. We will use a generic link factor that implements post hoc Bayesian model comparison and averaging (K. Friston & Penny, 2011; Hoeting, Madigan, Raftery, & Volinsky, 1999). Technically, equipping generative models of latent categorical states with the ability to handle continuous data means that one can categorize continuous data—and use posterior beliefs about categories as empirical priors for processing continuous data (e.g., time series). Clearly, this is something that the brain does all the time during perceptual categorization and action selection. For an introduction to Forney factor graphs, see Kschischang, Frey, & Loeliger (2001) and Loeliger (2002).

This paper comprises six sections. The first overviews active inference in terms of the (normative) imperative to minimize surprise, resolve uncertainty, and (implicitly) maximize model evidence. This section focuses on the selection of actions or policies (sequences of actions) that minimize expected free energy—and what this entails intuitively. Having established the basic premise that the brain engages in active (Bayesian) inference, we then turn to the generative models for which evidence is sought. The second section considers models where the states or causes of data are discrete or categorical in nature. In particular, it considers generative models based upon Markov decision processes, characterized in terms of Bayesian networks and Forney factor graphs. From these, we develop a putative (neural network) microcircuitry that could implement the requisite belief propagation. This section also takes the opportunity to distinguish between Bayesian inference and active inference by combining (Bayesian network and Forney factor) graphical formulations to show how the products of inference couple back to the process of generating (sensory) data, thereby enabling the brain to author the data or evidence that it accumulates. This section concludes by describing deep models that are composed of Markov decision processes, nested hierarchically over time.

The third section applies the same treatment to generative models with continuous states using a general formulation, based on generalized coordinates of motion (Friston, Stephan, Li, & Daunizeau, 2010). This treatment emphasizes the formal equivalence with belief propagation under discrete models. The fourth section considers generative models in which a Markov decision process is placed on top of a continuous state space model. This section deals with the special problem of how the two parts of the model are linked. The technical contribution of this paper is to link continuous states to discrete states through Bayesian model averages of discrete priors (over the causes of continuous dynamics). Conversely, the posterior probability density over these causes is converted into a discrete representation through Bayesian model comparison. The section concludes with a proposal for extrinsic (between-region) message passing in the brain that is consistent with the architecture of belief propagation under mixed generative models. In particular, we highlight the possible role of message passing between cortical and subcortical (basal ganglia and thalamic) systems. The fifth section illustrates belief propagation and the attendant process theory using simulations of (metaphorical) reading. This section shows how message passing works and clarifies notions like hidden states, using letters, words, and sentences. Crucially, inferences made about (discrete) words and sentences use (continuous) sensory data solicited by saccadic eye movements that accrue visual (and proprioceptive) input over time. We conclude with a brief section on the implications for context-sensitive connectivity that can be induced under the belief propagation scheme. We focus on the modulation of intrinsic excitability; specifically, the afferents to superficial pyramidal cells—and, as a more abstract level, the implications for self-organized criticality of the sort entailed by dynamic fluctuations in connectivity (Aertsen, Gerstein, Habib, & Palm, 1989; Allen et al., 2012; Baker et al., 2014; Breakspear, 2004).

All that follows is predicated on defining what the brain does or, more exactly, what properties it must possess to endure in a changing world. In this sense, the brain conforms to the imperatives for all sentient creatures; namely, to restrict itself to a limited number of attracting states (Friston, 2013). Mathematically, this can be cast as minimizing self-information or surprise (in information theoretic terms). Alternatively, this is equivalent to maximizing Bayesian model evidence; namely, the probability of sensory exchanges with the environment, under a model of how those sensations were caused. This is the essence of the free energy principle and its corollary—active inference—that can be neatly summarized as self-evidencing (Hohwy, 2016). Intuitively, self-evidencing means the brain can be described as inferring the causes of sensory samples while, at the same time, soliciting sensations that are the least surprising (e.g., not looking at the sun directly or maintaining thermoreceptor firing within a physiological range). Technically, this take on action and perception can be cast as minimizing a proxy for surprise; namely variational free energy. Crucially, active inference generalizes Bayesian inference in that the objective is not just to infer the latent or hidden states that cause sensations but to act in a way that will minimize expected surprise in the future. In information theory, expected surprise is known as entropy or uncertainty. This means that one can define optimal behavior as acting to resolve uncertainty (e.g., saccading to salient, or information rich, regimes of visual space or avoiding outcomes that are, a priori, costly or unattractive). In the same way that direct action and perception minimize surprise vicariously, through minimizing free energy, action can be specified in terms of policies that minimize the free energy expected when pursuing that policy.

In what follows, we will be less concerned with the pragmatic or utilitarian aspect of expected free energy and focus on the epistemic drive to explore salient regimes of the sensorium. We have previously addressed this epistemic foraging in terms of saccadic eye movements, using a generalized Bayesian filter as a model of neuronal dynamics (Friston, Adams, Perrinet, & Breakspear, 2012). In this paper, we reproduce the same sort of behavior but much more efficiently, using a generative model that entertains both discrete and continuous states. In brief, we will use a discrete state space model to generate empirical priors or predictions about where to look next, and a continuous state space model to implement those predictions, thereby garnering (visual) information that enables a constructivist explanation for visual samples: namely scene construction (Hassabis & Maguire, 2007; Mirza, Adams, Mathys, & Friston, 2016). The penultimate section presents simulations of reading to illustrate the use of deep generative models in active inference. However, first we consider the nature of generative models and the belief updating that they entail. In what follows, it will be useful to keep in mind the distinction between a true generative process in the real world and an agent’s generative model of that process. This distinction is important because active inference deals with how the generative model of a process and the process per se are coupled in a circular fashion to describe the perception-action cycle (Fuster, 2004; Tishby & Polani, 2010).

This section focuses on generative models of discrete outcomes caused by discrete states that cannot be observed directly (i.e., latent or hidden states). In brief, the unknown variables in these models correspond to states of the world that generate the outcomes of policies or sequences of actions. Note that policies have to be inferred. In other words, in active inference one has to infer what policy one is currently pursuing, where this inference can be biased by prior beliefs or preferences. It is these prior preferences that lend action a purposeful and goal-directed aspect.

Figure 1 describes the basic form of these generative models in complementary formats, and the implicit Bayesian belief updating following the observation of new (sensory) outcomes. The equations on the left specify the generative model in terms of a probability distribution over outcomes, states, and policies that can be expressed in terms of marginal densities or factors. These factors are conditional distributions that entail conditional dependencies, encoded by the edges in the Bayesian network on the upper right. The model in Figure 1 generates outcomes in the following way. First, a policy (i.e., action sequence) is selected at the highest level using a softmax function of the free energy expected under plausible policies. Sequences of hidden states are then generated using the probability transitions specified by the selected policy, which are encoded in B matrices. These encode probability transitions in terms of policy-specific categorical distributions. As the policy unfolds, the states generate probabilistic outcomes at each point in time. The likelihood of each outcome is encoded by A matrices, in terms of categorical distributions over outcomes, under each state.

The equivalent representation of this graphical model is shown as a Forney factor graph on the lower right. Here, the factors of the generative model (numbers in square boxes) now constitute the nodes and the (probability distribution over the) unknown states are associated with edges. The rules used to construct a factor graph are simple: The edge associated with each variable is connected to the factors in which it participates. If a variable appears in only one factor (e.g., policies), then the edge becomes a half-edge. If a variable appears in more than two factors (e.g., hidden states), then (copies of) the variable are associated with several edges that converge on a special node (labeled with “=”). Known or observed variables are usually denoted with small field squares. Note the formal similarity between the Bayesian network and the Forney factor graph; however, also note the differences. In addition to the movement of random variables to the edges, the edges are undirected in the Forney factor graph. This reflects the fact that messages are sent over edges in both directions. In this sense, the Forney factor graph provides a concise summary of the message passing implicit in Bayesian inference.

Heuristically, to perform inference on these graphs, one clamps the outputs to a particular (observed) value and passes messages from each node to each edge until (if necessary) convergence. The messages from node N to edge V, usually denoted by μ_N→V, comprise the sufficient statistics of the marginal probability distribution over the edge’s variable. These sufficient statistics (e.g., expectations) encode the requisite posterior probability. To illustrate the composition of messages during belief updating, we will illustrate the derivation of the first update (on the lower left of Figure 1) for expectations about hidden states.

The key aspect of this graph is that it discloses the messages that contribute to the posterior marginal over hidden states; here, conditioned on each policy. These constitute (forward: 2) messages from representations of the past, (backward: 3) messages from the future, and (likelihood: 4) messages from the outcome. Crucially, the past and future are represented at all times so that as new outcomes become available, with passage of time, more likelihood messages participate in the message passing, thereby providing more informed (approximate) posteriors. This effectively performs online data assimilation (mediated by forwarding messages) that is informed by prior beliefs concerning future outcomes (mediated by backward messages). Note that the policy is associated with a half-edge. This is because it appears in only one factor; namely, the probability distribution over policies based upon expected free energy G. Furthermore, the policy does not appear to participate in the message passing; however, we will see below that policy expectations play a key role, when we couple the message passing to the generative process—to complete an active inference scheme (and later when we consider the coupling between levels in hierarchical models). The reason the policy is not required for belief propagation among hidden state factors is that we have lumped together the hidden states s under each policy as a single variable (and the associated probability factors B) for clarity. This means that the message passing among the factors encoding hidden states proceeds in parallel for each policy, irrespective of how likely that policy is. Finally, note that the outcomes that inform the expected free energy are not the observed outcomes but predicted outcomes based upon expected states, under each policy (i.e., message 5).

Note that the Forney factor graph in Figure 1 posits separate messages for hidden states over time—and under each policy. This is consistent with what we know of neuronal representations; for example, distinct (place coded) representations of the past and future are implicit in the delay period activity shown in prefrontal cortical units during delayed matching to sample (Kojima & Goldman-Rakic, 1982); Friston, FitzGerald, et al. (2017) for a discussion. Furthermore, separable (place coded) representations of policies are ubiquitous in the brain; for example, salience maps (Bender, 1981; Zelinsky & Bisley, 2015) or the encoding of (overt or covert) saccadic eye movements in the superior colliculus (Müller, Philiastides, & Newsome, 2005; Shen, Valero, Day, & Paré, 2011).

Figure 2 combines Bayesian and Forney factor graphs to distinguish between the process generating outcomes and the concomitant inference that the outcomes induce. Crucially, the Bayesian network describing the generative process is not the Bayesian network describing the generative model, upon which the factor graph is based. In other words, in active inference the generative model and process are distinct. This is because the actual causes of data depend upon action and action depends upon inference (about policies). In other words, the end point of inference is a belief about policies that specify actions—and actions affect the transitions among the true states generating data. In short, the inference scheme effectively chooses the data it uses for inference. This means that the hidden policies do not actually exist; they are fictive constructs that are realized through action. This is an important part of active inference and shows how policies are coupled to the real world through action to complete the perception and action cycle. It also highlights the circular causality mediated by message passing: Messages flow from outcome nodes to a factor corresponding to expected free energy that determines the probability distribution over policies, thereby producing purposeful (epistemic and goal-directed) behavior. In summary, there are several useful insights into the computational architecture afforded by graphical representations of belief propagation. However, what does this tell us about the brain?

The neural network in Figure 3 tries to align the message passing in the Forney factor graph with quantitative studies of intrinsic connections among cortical layers (Thomson & Bannister, 2003).This (speculative) assignment allows one to talk about the functional anatomy of intrinsic connectivity in terms of belief propagation. In this example, state prediction error units (pink) have been assigned to granular layers (e.g., spiny stellate populations) that are in receipt of ascending sensory information (the likelihood message). These project to units in supragranular layers that represent (policy specific) expected states (cyan), which are averaged to form Bayesian model averages—associated with superficial pyramidal cells (red). The expected states then send forward (intrinsic) connections to units encoding expected outcomes in infragranular layers, which in turn excite outcome prediction errors (necessary to evaluate expected free energy; see Appendix 2: Friston, Parr, et al., 2017). This nicely captures the forward (generally excitatory) intrinsic connectivity from granular, to supragranular, to infragranular populations that characterize the canonical cortical microcircuit (Bastos et al., 2012; Douglas & Martin, 1991; Haeusler & Maass, 2007; Heinzle, Hepp, & Martin, 2007; Shipp, 2016). Note also that reciprocal (backward) intrinsic connections from the expected states to state prediction errors are inhibitory, suggesting that both excitatory and inhibitory interneurons in the supragranular layer encode (policy specific) expected states. Computationally, Equation 6 suggests this (interlaminar) connection is inhibitory because the last contribution (from expected states) to the prediction error is negative. Neurobiologically, this may correspond to a backward intrinsic pathway that is dominated by projections from inhibitory interneurons (Haeusler & Maass, 2007).

The particular formulation in Equation 6 distinguishes between the slower dynamics of populations encoding expectations of hidden states and the instantaneous responses of populations encoding prediction errors. This formulation leads to interesting hypotheses about the characteristic membrane time constants of spiny stellate cells encoding prediction errors, relative to pyramidal cells encoding expectations (e.g., Ballester-Rosado et al., 2010). Crucially, because prediction errors are a function of expectations and the rates of change of expectations are functions of prediction errors, one would expect to see the same sort of spectral differences in laminar-specific oscillations that have been previously discussed in the context of predictive coding (Bastos et al., 2012, 2015; Bosman et al., 2012).

This and subsequent neural networks should not be taken too seriously; they are offered as a starting point for refinement and deconstruction, based upon anatomy and neurophysiology. See (Shipp, 2016) for a nice example of this endeavor. The neural network above inherits a lot of its motivation from similar (more detailed) arguments about the laminar specificity of neuronal message passing in canonical microcircuits implied by predictive coding. Fuller accounts of the anatomical and neurophysiological evidence—upon which these arguments are based—can be found in (Adams, Shipp, & Friston, 2013), Bastos et al. (2012), Friston (2008), Mumford (1992), Shipp (2005, 2016), and (Shipp, Adams, & Friston, 2013). See (Whittington & Bogacz, 2017) for treatment that focuses on the role of intralaminar connectivity in learning and encoding uncertainty. One interesting component that the belief propagation scheme brings to the table (that is not in predictive coding; see Figures 7 and 10 below) is the encoding of outcome prediction errors in deep layers that send messages to (subcortical) nodes encoding expected free energy. This message passing could be mediated by corticostriatal projections, from layer 5 (and deep layer 3) pyramidal neurons, which are distributed in a patchy manner (Haber, 2016). We now move from local (intrinsic) message passing to consider the basic form of hierarchical message passing, of the sort that might be seen in cortical hierarchies.

The generative model in Figure 1 considers only a single timescale or temporal horizon specified by the depth of policies entertained. Clearly, the brain models the temporal succession of worldly states at multiple timescales, calling for hierarchical or deep models. An example is provided in Figure 4 that effectively composes a deep model by diverting some (or all) of the outputs of one model to specify the initial states of another (subordinate) model, with exactly the same form. The key aspect of this generative model is that state transitions proceed at different rates at different levels of the hierarchy. In other words, the transition from one hidden state to the next entails a sequence of transitions at the level below. This is a necessary consequence of conditioning the initial state at any level on the hidden states in the level above. Heuristically, this hierarchical model generates outcomes over nested timescales, like the second hand of a clock that completes a cycle for every tick of the minute hand that precesses more quickly than the hour hand. It is this particular construction that lends the generative model a deep temporal architecture. In other words, hidden states at higher levels contextualize transitions or trajectories of hidden states at lower levels to generate a deep narrative.

In terms of message passing, the equivalent Forney factor graph (Figure 4: lower right) shows that the message passing within each level of the model is conserved. The only difference is that messages are sent in both directions along the edge connecting the factor (D) representing the joint distribution over the initial state conditioned upon the state of the level above. These messages correspond to ascending and descending messages, respectively. The ascending message 6 effectively supplements the observations of the level above using the (Bayesian model) average of the states at the level below. Conversely, the descending message 2 plays a role of an empirical prior that induces expectations about the initial state of the level below, thereby contextualizing fast subordinate sequences within slower supraordinate narratives.

Crucially, the requisite Bayesian model averages depend on expected policies via message 1. In other words, the expected states that constitute descending priors are a weighted mixture of policy-specific expectations that, incidentally, mediate a Bayes optimal optimism bias (Friston et al., 2013; Sharot, Guitart-Masip, Korn, Chowdhury, & Dolan, 2012).This Bayesian model averaging lends expected policies a role above and beyond action selection. This can be seen from the Forney factor graph representation, which shows that messages are passed from the expected free energy node G to the initial state factor D. In short, policy expectations now exert a powerful influence over how successive hierarchical levels talk to each other. We will pursue this later from an anatomical perspective in terms of extrinsic connectivity and cortico–basal ganglia–thalamic loops. Before considering the implications for hierarchical architectures in the brain, we turn to the equivalent message passing for continuous variables, which transpires to be predictive coding (Rao & Ballard, 1999; Srinivasan, Laughlin, & Dubs, 1982).

This section rehearses the treatment of the previous section using models of continuous states. We adopt a slightly unusual formulation of continuous states that both generalizes established Bayesian filtering schemes and, happily, has a similar form to generative models for discrete states. This generalized form rests upon describing trajectories in generalized coordinates of motion. These are common in physics (e.g., position and momentum). Here, we consider motion to arbitrarily high order (i.e., location, velocity, acceleration, jerk). A key thing to bear in mind, when dealing with generalized motion, is that the mean of the generalized motion is the motion of the mean when, and only when, free energy is minimized. This corresponds to Hamilton’s principle of least action.

Figure 5 shows a generative model for a short sequence or trajectory described in terms of generalized motion. The upper panel (on the left) shows that an outcome is generated (with a static nonlinear mapping g) from the current state, with some random fluctuations; similarly for higher orders of motion. The motion of hidden states is a function (equation of motion or flow f) of the current state, plus random fluctuations. Higher-order motion is derived in a similar way using generalized equations of motion (in the upper left inset). The Bayesian network (on the upper right) has been written in a way that highlights its formal similarity with the equivalent network for discrete states (Figure 1). This entails lumping together the generalized hidden causes that are generated from a prior expectation. In this form, one can see that the probability transition matrices are replaced with generalized equations of motion, while the likelihood mapping becomes the static nonlinearity. The corresponding Forney factor graph is shown on the lower right, where we have introduced some special (local function) nodes corresponding to factors generating random fluctuations and their addition to predictions of observed trajectories and the flow of hidden states.

The ensuing generalized variational or Bayesian filtering scheme has several interesting special cases, including the (extended) Kalman (Bucy) filter, which falls out when we only consider generalized motion to first order. See Friston et al. (2010) for details. When expressed in terms of prediction errors, this generalized variational filtering corresponds to predictive coding (Rao & Ballard, 1999; Srinivasan et al., 1982) that has become an accepted metaphor for evidence accumulation in the brain. In terms of active inference, the minimization of free energy with respect to action or control states only has to consider the prediction errors on outcomes (because these are the only things that can be changed by action). This leads to the active inference scheme in Figure 6.

As with the discrete models, action couples back to the generative process through affect ing state transitions or flow. Note, as above, the generative process can be formally distinct from the generative model. This means the real equations of motion (denoted by boldface functions f) become functions of action. In this context, one can see how the (fictive) hidden causes in the generative model are replaced by (or supplemented with) action; that is a product of inference. The joint minimization of free energy—or maximization of model evidence—by action and perception rests upon the implicit closure of conditional dependencies between the process (world) and model (brain). See K. Friston (2011) and K. Friston, Mattout, and Kilner, (2011) for more details.

Figure 7 depicts a neural network that might implement the message passing in Figure 6. This proposal is based upon the anatomy of intrinsic and extrinsic connections described in Bastos et al. (2012). This figure provides the update dynamics for a hierarchical generalization of the generative model in Figure 6, where the outputs of a higher level now become the hidden causes of the level below. In this hierarchical setting, the prediction errors include prediction errors on both hidden causes and states. As with the model for discrete states, the prediction errors have been assigned to granular layers that receive sensory afferents and ascending prediction errors from lower levels in the hierarchy. Given that the message passing requires prediction errors on hidden causes to be passed to higher levels, one can assume they are encoded by superficial pyramidal cells; that is, the source of ascending extrinsic connections (Bastos et al., 2012; Felleman & Van Essen, 1991; Markov et al., 2013). Similarly, the sources of descending extrinsic connections are largely restricted to deep pyramidal cells that can be associated with expected hidden causes. The ensuing neural network is again remarkably consistent with the known microcircuitry of extrinsic and intrinsic connections, with a series of forward (intrinsic connections from granular, to supragranular, to infragranular layers) and reciprocal inhibitory connections (Thomson & Bannister, 2003). A general prediction of this computational architecture is that no pyramidal cell (early in cortical hierarchies) can be the source of both forward and backward connections. This follows from the segregation of neuronal populations encoding expectations and errors respectively, where error units send ascending projections to the granular layers of a higher area, while pyramidal cells encoding expectations project back to error units in lower areas. This putative law of extrinsic connectivity has some empirical support as reviewed in Shipp (2016).

The main difference between the microcircuits for discrete and continuous states is that the superficial pyramidal cells encode state expectations and prediction errors respectively. This is because in discrete belief propagation ascending and descending messages convey posteriors (i.e., expected states computed by the lower level) and empirical priors (i.e., the states predicted by the higher level) respectively; whereas, in predictive coding, they convey prediction errors and predictions respectively. Furthermore, the encoding of expected outcomes is not necessary for predictive coding. This is because we have replaced policies with hidden causes. One might ask how predictive coding can support epistemic and purposeful behavior in the absence of policies that underwrite behavior. The answer pursued below is that the hidden causes (of continuous state models) are provided by the policy-rich predictions of (discrete state) models. In the next section, these enriched predictions are considered in terms of message passing and salience.

This section considers the integration of discrete and continuous models, and what this implies for message passing in neuronal circuits. In brief, we will see that discrete outcomes select a specific model of continuous trajectories or dynamics—as specified by a prior over their hidden causes. Effectively, this generative model generates (discrete) sequences of short (continuous) trajectories defined in terms of their generalized motion. See Figure 8. Because state transitions occur discretely, the hidden causes generating dynamics switch periodically. This sort of model, if used by the brain, suggests the sensorium is constructed from discrete sequences of continuous dynamics (see also Linderman et al., 2016, for a recent machine learning perspective on this scenario). An obvious example here would be a sequence of saccadic eye movements, each providing a sample of the visual world and yet each constituted by carefully crafted oculomotor kinetics. In fact, this is an example that we will use below, where the discrete outcomes or priors on hidden causes specify a fixed-point attractor for proprioceptive (oculomotor) inference. In essence, this enables the discrete model to prescribe salient points of attraction for visual sampling.

In terms of belief propagation, Figure 9 shows that the descending messages comprise Bayesian model averages of predicted outcomes, while ascending messages from the lower, continuous, level of the hierarchical model are the posterior estimate of these outcomes, having sampled some continuous observations. In other words, the descending messages provide empirical priors over the dynamics of the lowest (continuous) level that returns the corresponding posterior distribution. This posterior distribution is interesting because it constitutes a Bayesian model comparison. This follows because we have treated each outcome as a particular model of dynamics, defined by plausible priors over hidden causes. Therefore, the posterior distribution over priors corresponds to the posterior probability for each model, given the continuous data at hand. From the point of view of the discrete model, each dynamic model corresponds to an outcome. From the point of view of the continuous level, each dynamic model corresponds to a particular prior on hidden causes.

In summary, the link factor that enables one to combine continuous and discrete (hierarchical) models corresponds to a distribution over models of dynamics—and mediates the transformation of descending model averages into ascending model posteriors—using Bayesian model averaging and comparison respectively. Effectively, this equips the associated belief propagation scheme with the ability to categorize continuous data into discrete categories and, in the context of the deep temporal models considered here, chunk continuous sensory flows into discrete sequential representations. We will exploit this faculty in simulations of reading below. However, first we consider the implications of this hierarchical generative model for extrinsic (hierarchical) message passing in the brain.

Figure 10 sketches an architecture that comprises three levels of a discrete hierarchical model and a single continuous level. In this neural network, we have focused on the extrinsic (between-region) connections that pass ascending prediction errors and expectations about subordinate states—and descending predictions (of initial discrete states or causes of dynamics). In virtue of assigning the sources of ascending messages to superficial pyramidal cells and the sources of descending messages to deep pyramidal cells, this recurrent extrinsic connectivity conforms to known neuroanatomy (Bastos et al., 2012; Felleman and Van Essen, 1991; Markov et al., 2013). An interesting exception here is the laminar specificity of higher-level (discrete) descending projections that arise in deep layers but target state prediction errors assigned to granular layers (as opposed to the more conventional super granular layers). Neuroanatomically, this may reflect the fact that laminar specificity is less pronounced in short-range extrinsic connections (Markov et al., 2013). An alternative perspective rests on the fact that higher (dysgranular) cortical areas often lack a distinct granular layer (Barbas, 2007; Barrett & Simmons, 2015), leading to the speculation that dysgranular cortex may engage in belief updating of categorical or discrete sort.

The belief propagation entailed by policy and action selection in Figure 10 is based upon the anatomy of cortico–basal ganglia–thalamic loops described in Jahanshahi, Obeso, Rothwell, and Obeso (2015). If one subscribes to this functional anatomy, the form of belief propagation suggests that competing low-level (motor executive) policies are evaluated in the putamen; intermediate (associative) policies in the caudate; and high-level (limbic) policies in the ventral striatum. These representations then send (inhibitory or GABAergic) projections to the globus pallidus that encodes the expected (selected) policy. These expectations are then com municated via thalamocortical projections to superficial layers encoding Bayesian model averages. From a neurophysiological perspective, the best candidate for the implicit averaging would be matrix thalamocortical circuits that “appear to be specialized for robust transmission over relatively extended periods, consistent with the sort of persistent activation observed during working memory and potentially applicable to state-dependent regulation of excitability” (Cruikshank et al., 2012, p. 17813). This implicit belief updating is consistent with invasive recordings in primates, which suggest an anteroposterior gradient of time constants (Kiebel, Daunizeau, & Friston, 2008; Murray et al., 2014). Note that the rather crude architecture in Figure 10 does not include continuous (predictive coding) message passing that might operate in lower hierarchical areas of the sensorimotor system. This means that there may be differences in terms of corticothalamic connections in prefrontal regions, compared with primary motor cortex, which has a distinct (agranular) laminar structure. See Shipp et al. (2013) for a more detailed discussion of these regionally specific differences.

The exchange of prior and posterior expectations about discrete outcomes between the categorical and continuous parts of the model have been assigned to corticothalamic loops, while the evaluation of expected free energy and subsequent expectations about policies have been associated with the cortical–basal ganglia–thalamic loops. An interesting aspect of this computational anatomy is that posterior beliefs about where to sample the world next are delivered from higher cortical areas (e.g., parietal cortex), where this salient sampling depends upon subcortical projections, informing empirical prior expectations about where to sample next. One could imagine these arising from the superior colliculus and/or pulvinar in a way that would be consistent with their role as a salience map (Robinson & Petersen, 1992; Veale, Hafed, & Yoshida, 2017). In short, sensory evidence garnered from the continuous level of the model is offered to higher levels in terms of posterior expectations about discrete outcomes. These high levels reciprocate with empirical priors that ensure the right sort of dynamic engagement with the world.

Clearly, there are many anatomical issues that have been ignored here, such as the distinction between direct and indirect pathways (Frank, 2005), the role of dopamine in modulating the precision of beliefs about policies (Friston, Schwartenbeck, et al., 2014), and so on. However, the basic architecture suggested by the above treatment speaks to the biological plausibility of belief propagation under the generative models. This concludes our theoretical treatment of belief propagation in the brain and the implications for intrinsic and extrinsic neuronal circuitry. The following section illustrates the sorts of behavior that emerge under this sort of architecture.

This section tries to demystify the notions of hidden causes, states, and policies by presenting a simulation of pictographic reading. This simulation is a bit abstract but serves to illustrate the accumulation of evidence over nested timescales—and the integration of discrete categorization with continuous oculomotor sampling of a visual world. Furthermore, it highlights the role of the discrete part of the model in guiding the sampling of a continuous sensorium. We have previously presented the discrete part in the context of scene construction and simulated reading (Mirza et al., 2016). In this paper, we focus on the integration of a Markov decision process model of visual search (Mirza et al., 2016) with a predictive coding model of saccadic eye movements (K. Friston et al., 2012), to produce a complete (mixed) model of evidence accumulation.

In brief, the generative model has two discrete levels. The highest level generates a sentence by sampling from one of six possibilities, where each sentence comprises four words. Given where the (synthetic) subject is currently looking, the sentence therefore specifies a word and the hidden states at the level below. These comprise different letters that can be located in quadrants of the visual field. Given the word and the quadrant currently fixated, the letter is specified uniquely. These hidden states now prescribe the dynamical model; namely, the attracting point of fixation and the content of visual input (i.e., a pictogram) that would be seen at that location. These dynamics are mediated simply by a continuous generative model, in which the center of fixation is attracted to a location prior, while the visual input is determined by the pictogram or letter at that location. Figure 11 provides a more detailed description of the generative model. The discrete parts of this model have been used previously to simulate reading. We will therefore concentrate on the implicit evidence accumulation and prescription of salient target locations for saccadic eye movements. Please see Mirza et al. (2016) for full details of the discrete model.

An interesting aspect of this generative model is that the world or visual scene is represented in terms of affordances; namely, the consequences of acting on—or sampling from—a visual scene. In other words, the generative model does not possess a “sketch pad” on which the objects that constitute a scene (and their spatial or metric relationships) are located. Conversely, the metric aspect of the scene is modeled in terms of “what would happen if I did this.” For example, an object (e.g., letter) is located to the right of another object because “this is the object that I would see if I look to the right.” Note that sensory attenuation ensures that visual impressions are only available after saccadic fixation. This means that the visual stream is composed of a succession of snapshots (that were generated in the following simulations using the same process assumed by the generative model).

To simulate reading, the equations in Figure 10 were integrated using 16 iterations for each time point at each discrete level. At the lowest continuous level, each saccade is assumed to take about 256 ms. (This is roughly the amount of time taken per iteration on a personal computer—to less than an order of magnitude.) This is the approximate frequency of saccadic eye movements, meaning that the simulations covered a few seconds of simulated time.

Figure 12 shows the behavior that results from integrating the message passing scheme described in Figure 10. Here, we focus on the eye movements within and between the four words (where the generative model assumed random fluctuations with a variance of one eighth). In this example, the subject looks at the first quadrant of the first word and sees a bird.She then looks to the lower right and sees nothing, confirming that this word is flee—by locating the cat on the upper right quadrant. The subject is now fairly confident that the sentence has to be the first or fourth sentence—that both have the same words in the second and third positions. This enables her to quickly scan the subsequent two words (with single saccades to resolve any residual uncertainty) and focus on the last word that disambiguates between two plausible sentences. Here, she discovers seeds on the second saccade and empty quadrants off the diagonal. At this point, residual uncertainty about the sentence is resolved and the subject infers the correct sentence.

An example of the evidence accumulation during a single saccade is provided in Figure 13. In this example, the subject looks from the central fixation to the upper left quadrant to see a cat. The concomitant action is shown as a simulated electroculogram in the middle left panel, with balanced movement in the horizontal and vertical directions. The corresponding visual input is shown for four consecutive points in time (i.e., every five time steps, where each of the 25 time steps of the continuous integration scheme corresponds roughly to 10 ms). Note that the luminance contrast increases as the center of gaze approaches the target location (specified by the empirical prior from the discrete part of the model). This implements a simple form of sensory attenuation. In other words, it precludes precise visual information during eye movement, such that high-contrast information is only available during fixation. Here, the sensory attenuation is implemented by the modulation of the visual contrast as a Gaussian function of the distance between the target and the current center of gaze (see Figure 11). This is quite a crude way of modeling sensory attenuation; however, it is consistent with the fact that we do not experience optic flow during saccadic eye movements.

The lower panels of Figure 13 show the evidence accumulation during the continuous saccadic sampling in terms of the posterior probability under the four alternative visual (what) outcomes (lower left) and the four proprioceptive (where) outcomes (lower right). In this instance, the implicit model comparison concludes that a cat is located in the first quadrant. Note that the posterior probability for the target location is definitive from the onset of the saccade (by virtue of the precise empirical priors from the level above). In contrast, imprecise (empirical prior) beliefs about the visual content take more time to resolve into precise (posterior) beliefs as the location priors are realized via action. This is a nice example of multimodal sensory integration and evidence accumulation. Namely, precise beliefs about the where state of the world are used to resolve uncertainty about the what aspects in a way that speaks directly to the epistemic affordance that underlies salience.

Figure 14 shows the simulated neuronal responses underlying the successive accumulation of evidence at discrete levels of the hierarchical model (cf. Huk & Shadlen, 2005). The neuro physiological interpretations of these results appeal to Equation 6, where expectations are encoded by the firing rates of principal cells and fluctuations in transmembrane potential are driven by prediction errors. A more detailed discussion of how the underlying belief propagation translates into neurophysiology can be found in K. Friston, FitzGerald, et al. (2017).

Under the scheduling used in these simulations, higher-level expectations wait until lower-level updates have terminated and, reciprocally, lower-level updates are suspended until belief updating in the higher level has been completed. This means the expectations are sustained at the higher level, while the lower level gathers information with a saccade or two. The upper two panels show the (Bayesian model) averages of expectations about hidden states encoding the sentence (upper panel) and word (second panel). The lower two panels show simulated electrophysiological and behavioral responses respectively (reproduced from Figure 12). The key thing to note here is the progressive resolution of uncertainty at both levels—and on different timescales. Posterior expectations about the word fluctuate quickly with each successive visual sample, terminating when the subject is sufficiently confident about what she is sampling. The posterior expectations are then assimilated at the highest level, on a slower timescale, to resolve uncertainty about the sentence in play. Here, the subject correctly infers, on the last saccade of the last word, that the first sentence generated the stimuli. These images or raster plots can be thought of in terms of firing rates in superficial pyramidal populations encoding the Bayesian model averages (see Figure 10). The resulting patterns of firing show a marked resemblance to presaccadic delay period activity in the prefrontal cortex (Funahashi, 2014). The corresponding (simulated) local field potentials are shown below the raster plots. These are just band-pass filtered (between 4 and 32 Hz) versions of the spike rates that can be interpreted in terms of depolarization. The fast and frequent (red) evoked responses correspond to the Bayesian model averages (pertaining to the three possible words) at the first level, while the interspersed and less frequent transients (blue) correspond to updates at the highest level (over six sentences). The resulting succession of local field potentials or event-related potentials again look remarkably similar to empirical responses in inferotemporal cortex during active vision (Purpura, Kalik, & Schiff, 2003). Although not pursued here, one can perform time frequency analyses on these responses to disclose interesting phenomena such as theta gamma coupling (entailed by the fast updating within saccade that repeats every 250 ms between saccades). In summary, the belief propagation mandated by the computational architecture of the sort shown in Figure 10 leads to a scheduling of message passing that is similar to empirical perisaccadic neuronal responses in terms of both unit activity and event-related potentials.

In the previous section, we highlighted the biological plausibility of belief propagation based upon deep temporal models. In this section, this biological plausibility is further endorsed by reproducing electrophysiological phenomena such as perisaccadic delay period firing activity and local field potentials. Furthermore, these simulations have a high degree of face validity in terms of saccadic eye movements during reading (Rayner, 1978, 2009).

We have derived a computational architecture for the brain based upon belief propagation and graphical representations of generative models. This formulation of functional integration offers both a normative and a process theory. It is normative in the sense that there is a clearly defined objective function; namely, variational free energy. An attendant process theory can be derived easily by formulating neuronal dynamics as a gradient descent on this proxy for surprise or (negative) Bayesian model evidence. The ensuing architecture and (neuronal) message passing offers generality at a number of levels. These include deep generative models based on a mixture of categorical states and continuous variables that generate sequences and dynamics. From an algorithmic perspective, we have focused on the link functions or factors that enable categorical representations to talk to representations of continuous quantities, such as position and luminance contrast. One might ask how all this helps us understand the nature of dynamic connectivity in the brain. Clearly, there are an enormous number of anatomical and physiological predictions that follow from the sort of process theory described in this paper; these range from the macroscopic hierarchical organization of cortical areas in the brain to the details of canonical microcircuits (e.g., Adams et al., 2013; Bastos et al., 2012; Friston, 2008; Mumford, 1992; Shipp, 2016). Here, we will focus on two themes: first, the implications for connectivity dynamics within cortical microcircuits and, second, a more abstract consideration of global dynamics in terms of self-organized criticality.

The update equations for both continuous and discrete belief propagation speak immediately to state- or activity-dependent changes in neuronal coupling. Interestingly, both highlight the importance of state-dependent changes in the connectivity of superficial pyramidal cells in supragranular cortical layers. This conclusion rests on the following observations.

Figure 2 suggests that the key modulation of intrinsic (cortical) connectivity is mediated by policy expectations. These implement the Bayesian model averaging (over policy-specific estimates of states) during state estimation. Physiologically, this means that the coupling between policy-specific states (here assigned to supragranular interneurons) and the Bayesian model averages (here assigned to superficial pyramidal cells) is a key locus of dynamic connectivity. This suggests that fluctuations in the excitability of superficial pyramidal cells are a hallmark of belief propagation under the discrete process theory on offer.

These conclusions are potentially important from the point of view of empirical studies. For example, it suggests that dynamic causal modeling of condition-specific, context-sensitive, effective connectivity should focus on intrinsic connectivity; particularly, connections involving superficial pyramidal cells that are the source of forward extrinsic (between region) connections in the brain (e.g., Auksztulewicz & Friston, 2015; Brown & Friston, 2012; Fogelson, Litvak, Peled, Fernandez-del Olmo, & Friston, 2014; Pinotsis et al., 2014). Indeed, several large-scale neuronal simulations speak to the potential importance of intrinsic excitability (or excitation-inhibition balance) in setting the tone for—and modulating—cortical interactions (Gilson, Moreno-Bote, Ponce-Alvarez, Ritter, & Deco, 2016; Roy et al., 2014).

Clearly, a focus on intrinsic excitability is important from a neurophysiological and pharmacological perspective. This follows from the fact that the postsynaptic gain or excitability of superficial pyramidal cells depends upon many neuromodulatory mechanisms. These include synchronous gain (Chawla, Lumer, & Friston, 1999) that is thought to be mediated by interactions with inhibitory interneurons that are, themselves, replete with voltage-sensitive NMDA receptors (Lee et al., 2013; Sohal et al., 2009). Not only are these mechanisms heavily implicated in things like attentional modulation (Fries, Reynolds, Rorie, & Desimone, 2001), they are also targeted by most psychotropic drugs and (endogenous) ascending modulatory neurotransmitter systems (Dayan, 2012). This focus—afforded by computational considerations—deals with a particular aspect of microcircuitry and neuromodulation. Can we say anything about whole-brain dynamics?

Although the generative model changes from application to application, the belief updates described in this paper are generic and can be implemented using standard routines (here spm_MDP_VB_X.m and spm_ADEM.m). These routines are available as Matlab code in the SPM academic software: http://www.fil.ion.ucl.ac.uk/spm/software. The simulations in this paper can be reproduced (and customized) via a graphical user interface by typing in >> DEM and selecting the Mixed models demo.

Karl Friston: Conceptualization; Formal analysis; Writing – original draft. Thomas Parr: Conceptualization; Formal analysis; Writing – review & editing. Bert de Vries: Conceptualization; Formal analysis; Writing – review & editing.

KJF is funded by the Wellcome Trust (Ref: 088130/Z/09/Z). TP is supported by the Rosetrees Trust (Award number: 173346).

We would like to thank our two anonymous referees for helpful guidance in presenting this work.

Generative model:

Or forward model; a probabilistic mapping from causes to observed consequences (data). It is usually specified in terms of the likelihood of getting some data given their causes (parameters of a model) and priors on the parameters.

Generalized coordinates of motion:

Cover the value of a variable, its motion, acceleration, jerk, and higher orders of motion. A point in generalized coordinates corresponds to a path or trajectory over time.

Factor graph:

A bipartite graph (where two distinct sets of nodes are connected by edges) representing the factorization of a function, usually a probability distribution function. Formulating a Bayesian network or model as a factor graph enables the efficient computation of marginal distributions, through the sum-product algorithm.

Bayesian network:

A Bayes network, model, or belief network is a probabilistic graphical model that represents a set of random variables and their conditional dependencies via a directed acyclic graph.

Free energy:

An information theory measure that bounds (is greater than) the surprise on sampling some data, given a generative model.

Empirical prior:

Priors that are induced by hierarchical models; they provide constraints on the recognition density in the usual way but depend on the data.

Surprise:

Surprisal or self-information is the negative log probability of an outcome. An improbable outcome is therefore surprising.

Prior:

The probability distribution or density on the causes of data that encode beliefs about those causes prior to observing the data.

Conditional density:

Or posterior density; the probability distribution of causes or model parameters, given some data; that is, a probabilistic mapping from observed data (consequences) to causes.

(Kullback-Leibler) Divergence:

Information divergence, information gain, or relative entropy is a noncommutative measure of the difference between two probability distributions.

Gradient descent:

An optimization scheme that finds a minimum of a function by changing its arguments in proportion to the negative of the gradient of the function at the current value.

Precision:

In general statistical usage, the inverse variance or dispersion of a random variable. The precision matrix of several variables is also called a concentration matrix. It quantifies the degree of certainty about the variables.