Bounded Rational Decision Networks With Belief Propagation

A fundamental organizing principle in any complex system is modularity. Whether it is cells coming together to form organisms, diverse neurons comprising brains, or humans establishing relationships and creating firms and societies, the shared characteristic is that organized groups have the capacity to accomplish tasks that are beyond the reach of any single individual. This idea was captured and studied extensively in a wide variety of fields, including artificial intelligence (Amer & Maul, 2019; Ellefsen et al., 2015), cognitive science (Fodor, 1983; Katahira et al., 2012), psychology (Bechtel, 2003; Samuels, 2012), management theory (Langlois, 2002), and evolutionary biology (Constantino & Daoutidis, 2019). Organized group behavior is often modeled in terms of a network of decision-making units that operate with constrained resources while optimizing a global objective through local interactions and communication. The global objective is usually specified by a shared utility function, sometimes supplemented with a cost function that captures information processing resources, thus forcing a trade-off between maximizing utility and minimizing cost.

Previously, we have studied networks of bounded rational information processing decision makers that implement such a trade-off between a global utility function and a local information processing cost in each decision node. So far we have focused on strictly hierarchical models with feedforward information flow (Genewein et al., 2015; Gottwald & Braun, 2019). To capture the full potential of such systems, however, we need to allow for recurrent communication protocols, which also requires a new mechanism that allows for the optimization of cyclical communication paths. In this article, we study how far belief propagation algorithms can be adapted to serve such a purpose. We consider an arbitrary network of bounded rational decision-making agents collaborating to solve a problem defined by a single utility function. Each agent operates autonomously, focusing on solving a local problem that contributes to the overarching task. However, as the whole system of agents optimizes a global utility function, information about other agents has to be transferred throughout the network. To this end, we combine the advantages of the belief propagation algorithm with the bounded rationality framework.

Belief propagation is a message-passing algorithm for inference in graphical models that exploits the graphical model structure to determine the marginal distributions for all the unobserved variables in the model, given the values of all observed variables (Pearl, 1988, 2013; Yedidia et al., 2003, 2005; Rehn & Maltoni, 2014; Steimer et al., 2009; Straszak & Vishnoi, 2019; Wainwright & Jordan, 2008). Belief propagation has found a wide range of applications. It is applied in computer vision for image segmentation and object recognition (Yan et al., 2023; Carbonetto et al., 2004), bioinformatics for protein structure prediction and network analysis of biological data (Soni et al., 2010; Isci et al., 2013), and social network analysis for identifying communities and influential nodes (Savage et al., 2014; Chunaev, 2020). In the context of bounded rationality, we might think of inference as the process of determining the probability distributions required for action sampling.

One central feature of belief propagation is its widespread applicability, including different types of graphical models, and its natural extension from tree graphs to graphs containing cycles. Cyclic graphs have gained significant attention in various research areas due to their ability to capture complex dependencies and dynamic relationships. In the field of machine learning, recurrent neural networks have emerged as powerful models for sequential data processing, utilizing cyclic connections to propagate information through time steps (Sak et al., 2014). This has paved the way for advancements in natural language processing, speech recognition, and time series analysis (Hongmei et al., 2013; Schmidt & Murphy, 2012). These models leverage cyclic connections to model contextual dependencies and improve accuracy. The exploration of cyclic graphs in various research domains continues to drive innovation and facilitate the development of more sophisticated models.

In what follows, we provide a short summary of related work in section 2 and an overview of the technicalities of bounded rational decision making in section 3, starting from preliminary comments on the (bounded) rational decision-making problem up until the organization of multiple decision-making units into multistep feedforward decision-making architectures. We also provide a description of a belief propagation algorithm, namely, the sum-product algorithm, and explain how we weave this algorithm into the bounded rational information processing framework to increase the efficiency of an otherwise often intractable update rule. Section 4 then generalizes this update rule to arbitrary graphical models, dismissing the necessity of a strictly feedforward information flow and allowing for repeated correspondence between decision nodes. In section 5, we give examples for the different information processing methods described in the previous section and provide evidence that decision-making processes containing loops outperform corresponding feedforward structures on various tasks, especially when information-theoretic resources for the decision makers involved are sparse. In section 6, we discuss our findings, and section 7 concludes the article.

One prominent approach that deals with utility-information trade-offs in complex decision networks is bounded rationality. Herbert Simon (1943, 1955) argued that bounded rational individuals face information-processing constraints when making decisions that can result in deviations from models of perfect rationality. Simon’s work on bounded rationality catalyzed interdisciplinary studies in economics (Visco & Zevi, 2020), management (Schiliro, 2013, 2012), psychology (Viale et al., 2023), and sociology (Bögenhold, 2021), revealing how limited information processing capabilities affect decision making. It also influenced the development of algorithms in computer science and artificial intelligence to mimic human decision making in areas like automated reasoning and machine learning (Hüllermeier, 2021).

Information-theoretic notions of bounded rationality (Ortega & Braun, 2013, Ortega et al., 2015; Mattsson & Weibull, 2002; Sims, 2003; Tishby & Polani, 2010; Friston, 2010; Still, 2009; Todorov, 2009; Bhui et al., 2021; Kappen et al., 2012; Wolpert, 2004; Leibfried & Braun, 2016; McKelvey & Palfrey, 1995) generalize the maximum expected utility principle by including an additive computational cost in the shape of a Kullback-Leibler divergence to measure deviations from prior behavior. If the prior is optimally chosen, then this cost is the mutual information between input and output. The resulting behavior trades off expected utility and computational cost, as represented by the optimal posterior taking the form of a Boltzmann distribution with inverse temperature parameter, interpolating between a completely rational decision maker (low temperature) and a decision maker that does not deviate from its prior behavior (high temperature).

A common way to deal with complex tasks is splitting them up into multiple simpler subtasks. In the realm of artificial intelligence and computing, multiple approaches have emerged from this idea to solve intricate problems. Multiagent optimization harnesses the collective intelligence of multiple autonomous agents that collaborate to find best solutions, where each agent typically possesses its own objectives and capabilities (Cerquides et al., 2014; Lobel et al., 2011; Terelius et al., 2011). Communication between agents allows for shared information and coordination needed for overcoming problems arising from limited knowledge about the overall system (Shirado & Christakis, 2017).

The approach of decentralized control distributes decision-making authority across various different agents, ensuring a resilient and adaptive system (Bakule, 2008; Yan et al., 2014; Schilling et al., 2021). Each agent in a decentralized system acts autonomously based on local information without relying on a central source of information or utility. Another approach that leverages decentralization, federated machine learning, consists of a method to train a model across multiple decentralized devices while keeping data localized (Yang et al., 2019; Wang et al., 2022). In a recent publication, Friston et al. (2023) extended the federated learning framework by introducing the concept of belief sharing based on active inference among agents and showed how communication enhances the collaborative learning process. Multiagent active inference was also previously used to describe the relationship between individual and collective inference in multiagent systems (Heins et al., 2022).

In the field of game theory, coordination games model interactions between players that benefit from aligning their choices with one another. In such games, communication is often a key ingredient for successful cooperation as many experimental studies have pointed out (Cooper et al., 1992; Cooper, 1999; Cason et al., 2012). One solution concept used in context of coordination games is rationalizability. It is based on rationality and the common belief in rationality and allows for uncertainty or incomplete information about the actions of other players (Bergemann & Morris, 2017). Furthermore, it was examined to what extent players can find best responses to previous actions played by other players with whom they were connected within a network (Jackson & Watts, 2002; Watts, 2001).

Classically, inference is applied to unobserved state-like random variables (considered hidden causes of observations), and actions are treated as (nonprobabilistic) model parameters. However, in the more recent literature, actions are often treated as random variables themselves, effectively transforming influence diagrams into Bayesian networks. In control as inference (Toussaint & Storkey, 2006; Kappen et al., 2012; Todorov, 2008; Toussaint, 2009; Levine, 2018), the analogous treatment of actions and state variables is also applied to inference, which is not only performed over state variables but also over actions. This allows the application of a vast amount of available inference techniques, such as exact Bayesian inference using conjugate priors, approximate inference using variational free energy optimization, and other methods (Levine, 2018). Formulating decision making as an inference problem also enables robust, adaptive, and probabilistic modeling (Shachter & Peot, 1992), allowing for uncertainty quantification (Bratvold et al., 2010) and the incorporation of prior knowledge (Huang et al., 2012). Moreover, approximate variational inference over actions is used in active inference and the (variational) free energy principle (Friston et al., 2017; Schwöbel et al., 2018; Millidge et al., 2021; Mitchell et al., 2019; Solway & Botvinick, 2012; Parr et al., 2019) to discuss the biological plausibility of inference mechanisms.

Note that while the variational free energy over action distributions and the free energy used in information-theoretic bounded rationality are formally equivalent under certain conditions (see section 6), they have different use cases and therefore appear in different scenarios: the latter is the result of trading-off performance (usually measured in terms of expected utility) and informational costs, whereas the former, variational free energy, is used in variational inference to approximate Bayes posteriors (see Gottwald & Braun, 2020; Gershman, 2019 for a detailed comparison).

Message-passing algorithms, such as the classic belief propagation algorithm by Pearl (1988), are computational tools leveraging the structure of the underlying Bayesian network to efficiently perform inference, both for exact inference in tree-like graphs, as well as for approximate inference in graphs with loops (Yedidia et al., 2003, 2005).

Therefore, when performing inference over actions, it has also been shown to be a valuable tool for control as inference (Toussaint, 2009; Levine, 2018). Additionally, such message-passing schemes are examined for their potential to serve as inference descriptions in biological networks as an attempt to explain neural processing (Friston et al., 2017; Schwöbel et al., 2018; Parr et al., 2019).

Local update rules and local interactions between units form the basis of some fundamental energy-based neural network models like Hopfield networks (Hopfield, 1982) and Boltzmann machines (Ackley et al., 1985). In classical and modern Hopfield networks, the updated state of a neuron depends on the states of its neighbors, enabling the storage and retrieval of patterns with low error (Tolmachev & Manton, 2020; Millidge et al., 2022), parameter estimation (Fazzino et al., 2021), and new ways of convolutional neural networks (Miconi, 2021) and deep learning (Ramsauer et al., 2020). Boltzmann machines can be seen as the stochastic counterpart of Hopfield networks as their global energy is identical in form to that of Hopfield networks and their update to each unit depends on probability distributions associated with their neighbors (Agliari et al., 2013; Osogami, 2017; Ota & Karakida, 2023).

In this section we give a brief motivation and overview of previous formulations for bounded rational decision making with information constraints. This will provide the conceptual framework and the corresponding terminology required for section 4.

In the following, we consider a decision maker to be a mechanism of at least one (decision-making) unit that observes an input, adapts to the observation, and reacts accordingly to the best of its ability. In the case of a decision maker $X$⁠, this mechanism can be described by specifying its observation $o∈O$⁠, its possible actions $x∈X$⁠, and the posterior probabilities $P(x|o):=P(X=x|O=o)$ of choosing action $x$ given observation $o$ (see Figure 1 for an illustration). We refer to the probability $p(x):=p(X=x)$ as the decision maker’s prior (choice) probability over actions, which can be thought of as the probability of making a blind guess for an action should the observation not be available. The utility function $U:O×X→R$ is a measure of value or preference and assigns a real-valued scalar to each observation-action pair.

Unless stated otherwise, we use upper-case letters $O,X,X1,X2,⋯$ to denote random variables, which are assumed to take values in finite sets $O,X,X1,X2,⋯$ . For simplicity, the decision maker who decides about random variable $X$ is also simply referred to as $X$⁠. The upper-case letter $P$ is used for posterior probability distributions, the lower-case letter $p$ is used for joint and prior distributions, and $PX$ is used for the set of all probability distributions defined on the set $X$⁠.

Consider a utility function $U$ whose value $U(o,x2)$ depends on observation $o∈O$ and action $x2∈X2$⁠. In the absence of further constraints, the optimal choice probability $P(X2|O)$ concentrates its probability mass on those actions $x2$ that maximize the utility for a given observation $o$⁠. In the deterministic case, we could simply devise an optimal mapping $x2=f(o)$ that assigns to each observation $o$ an action $x2$ that maximizes the utility over the space of possible actions $X2$⁠.

In the following, we are interested in distributed decision-making systems consisting of multiple decision-making units. For example, if we get just one more decision maker $X1$ involved in the process, we could form a serial chain of decision makers $O→X1→X2$⁠. Without further constraints, adding a middleman does not add any capabilities and, in fact, can only make matters worse due to the data processing inequality, as $P(X2|O)$ can already represent any mapping from input to output. However, if we restrict the class of permissible mappings each individual decision maker can implement, we can gain representational power, as is well known in the case of multilayered feedforward networks.

If we were to allow for multiple duplicates of the decision maker $X2$⁠, we could also use an additional decision maker $X1$ as a selector or indicator that assigns different subsets of observations $o$ to different instantiations of the decision maker $X2$⁠. Each of the instantiations could then be trained separately and specialize for their subset of observations. In such a parallel information processing architecture, we would essentially end up with a mixture-of-expert system that can store diverse prior knowledge through different expert decision makers.

In this article, we are particularly interested in distributed decision-making systems with recurrencies or loops. A simple example consisting of two decision makers could be $O→X1↔X2$⁠, where we allow for a recurrent sequence of updates between $X1$ and $X2$ with distributions $P(X1|O,X2)$ and $P(X2|X1)$⁠. Such recurrent updating is reminiscent of alternating optimization schemes, where subsets of parameters are optimized in an alternating fashion, or Gibbs sampling, where we repetitively sample from conditional distributions to generate samples from a joint distribution (Geman & Geman, 1984). The underlying reason that repetitive sampling can lead to improved performance is that the partial optimization and sampling processes that can be achieved by a single information processing step are not independent of the other partial information processing steps; otherwise, a single forward sweep might be sufficient. Thus, if there is no single agent that is powerful enough to jointly optimize, multiple agents that are only capable of partial optimization have to alternate. In the following, we are interested in exploring the power of recurrent information flow in the context of bounded rational decision-making networks by considering two scenarios. In the first scenario, we assume that all decision nodes are optimally equilibrated at each update step. Accordingly, when a node is part of a loop in the decision network, the same node is updated multiple times and will be represented by a different distribution after each update. In the second scenario, we consider a continual updating scheme, where each decision node is represented by the latest update resulting from the previous update.

The parameter $β∈(0,∞)$ interpolates between a nonadaptive decision maker whose posterior probability matches its prior probability for every observation (⁠$β→0$⁠) and a fully rational decision maker (⁠$β→∞$⁠) that always selects the action with highest expected utility. Hence, whenever $U(o,x)$ has a unique maximum, a rational agent’s posterior $P(·|o)$ is a Dirac distribution $δx*$⁠, assigning zero probability to all actions different from an optimal action $x*∈argmaxxU(o,x)$⁠. For every $β∈(0,∞)$⁠, optimizing the objective 3.1 forces a trade-off between the expected utility and the transformation cost given by the relative entropy $DKL(P||q)$⁠. This type of optimization objective has the form of a free energy that arises from constraints (Genewein et al., 2015; Gottwald & Braun, 2019, 2020).

Example. Imagine a teacher asking a student a question in an exam. For simplicity, there are 10 different possible questions that can roughly be categorized into two topics. For each question, there is a single best response that is rewarded with the best grade; all other responses that match at least the general topic get lower grades, and answers that do not even match the topic get the worst grades. In this scenario, the question asked by the teacher represents the observation, the student is the decision maker who has to find a good answer, and the student’s preparation can be regarded as the procedure of information processing. The $β$ parameter then describes the student’s capabilities, where a higher value of $β$ corresponds to a better student. Figure 1 illustrates this example using a graphical model (see Figure 1a) to describe student $X$’s answer depending on the question $o$ asked by the teacher. Figure 1b is an exemplary utility function where high utility (dark blue) corresponds to better grades and low utility (white, light blue) corresponds to lower grades. Figure 1c displays the average utility gained by students with a certain $β$ parameter and Figures 1d and 1e show the posterior probability of the responses of a student with low and a student with high beta.

The notion of a single decision maker from the previous section can be generalized to a multistep decision-making system by allowing multiple decision makers to be involved in the decision-making process (see Figure 2 for an example). Again, starting with an observation $o∈O$ and resulting in an action $x∈X$⁠, the decision-making process is decomposed into multiple intermediate steps. Each intermediate step consists of a new decision maker $Xi$⁠, and the multistep system is characterized by the connections between the different decision makers as given by the prior and posterior models.

Following Gottwald and Braun (2019), variables in the multistep decision-making process are labeled according to the sequence of information flow, that is, the decision-making system is characterized by a set of random variables $XI={X0,...,XN}$⁠, such that $Xj$ can obtain information about $Xi$ only if $j>i$⁠. Thus, $X0$ does not depend on the output of any other decision maker and none of the decision makers depend on the output of $XN$⁠, meaning $X0$ corresponds to the observation $O$ and $XN$ corresponds to the action output $X$ of the system. In other words, there is a (not necessarily unique) linear ordering of the variables involved in the decision-making process reflecting the information flow between the decision makers, which we discuss below.

A specific multistep architecture can be visualized by a directed graph, such that its nodes correspond to the decision makers involved in the decision-making process, its arcs describe the dependencies, and there is a topological sort of its nodes that corresponds to the linear ordering of the decision makers given by the information flow. In other words, in this topological sort, the graph is traversed in a way that ensures each node is encountered only after all its dependencies have been visited. Although generally the topological sort of a directed graph is not unique, we describe the decision-making process by defining prior and posterior models that correspond to one viable topological sort.

In the following, we use dashed arrows for prior-selector parent nodes and solid arrows for input nodes (see Figure 2 for examples and Figure 9 for an exemplary illustration of the interaction between prior-selector and input nodes). Moreover, as we allow the utility function $U:XIU→R$ to depend on an arbitrary subset of nodes involved in the decision-making process, whose indices we collect in the set denoted by $IU$⁠, we differentiate between utility nodes and intermediate nodes by coloring the nodes in the graphs blue and white, respectively. Intermediate nodes serve as hidden signals that filter information for subsequent nodes.

In general, we can say that at every time step, a decision maker $Xi$ processes incoming signals if its predecessors $XA(i)$ have processed their corresponding incoming signals. The time ordering of information processing in a multi step decision-making process will play an important role in section 4, where we generalize the feedforward graphs from Gottwald and Braun (2019) summarized above to graphs with loops.

This section serves as a brief summary of belief propagation with the sum-product algorithm, as we will incorporate a variant of this message-passing algorithm into the Blahut-Arimoto iteration from the previous section.

This setup enables a message-passing scheme, known as the sum-product algorithm, involving two main operations: summing and multiplying. In the first step, the algorithm computes the sum of products of incoming messages for each node in the graph, which gives the relative probability that the node is in a specific state, and in the second step, the node sends messages to its neighboring nodes based on that relative probability. The process is repeated until convergence is achieved, and the final messages can then be used for inference (see appendix A.2 for details).

In the following section, we give an overview of how to modify a factor graph to solve a series of different inference tasks.

Given a factor graph $F$ that represents the joint probability distribution of a set of random variables $p(XI)=∏j=1MfjXIj$⁠, the belief propagation algorithm can be used to find beliefs $b(Xi)$ about marginal distributions of each variable as discussed above (see appendix A.2 for a more detailed description). Additionally, whenever a set of variables $XIj$ appears together as input to a factor node $fj$⁠, the belief $b(XIj)$ can be determined by the normalized product of all incoming messages to the factor and the corresponding factor itself.

In order to obtain beliefs that approximate conditional probability distributions, for example, $p(Xi|Xj=xj)$⁠, the factor graph can be modified by adding a factor node connected to the variable node $Xj$ whose potential function consists of an indicator function fixing $Xj=xj$⁠. Similarly, in order to obtain beliefs about joint distributions $p(XJ)$ of an arbitrary subset of variables, a new factor node with constant potential function that is connected to all variable nodes $Xi$ with $i∈J$ can be added to the factor graph. Running the belief propagation algorithm on such modified factor graphs results in beliefs about conditional or joint distributions, respectively.

In what follows, we describe how we connect and combine the different factor graph modifications for an efficient inference mechanism for multistep decision-making systems like the ones in the previous section.

The two places where probabilistic inference is necessary when calculating the optimal posteriors in section 3.3 are the optimal priors 3.10 and the conditional probabilities for the expectations when determining the effective utilities (see equation 3.8). Hence, in order to use message passing, we transform the graphical model underlying the decision-making process into a factor graph and add modifications depending on the inference task (see appendix A.4).

We now present our main contributions. The prior and posterior models for the multistep decision-making systems studied in previous work (Gottwald & Braun, 2019) and discussed in section 3.3 can be represented by a directed graph where only leaves and roots of the graphical model are endpoints of the decision-making process, and, crucially, that communication between nodes happens in a feedforward manner through the network, where node $X$ can obtain information about another node $Y$ only if there is a directed path from $Y$ to $X$⁠. This description therefore does not include back-and-forth communication of decision makers, where information can flow in arbitrary directions.

In the following, we loosen the restrictions on the prior and posterior models such that the corresponding graphs may contain cycles and feedback loops, analogous to how the belief propagation algorithm is used on graphs containing cycles without convergence or correctness guarantees. As this removes the topological sort condition, the prior and posterior models are no longer sufficient to define the decision-making process as repeated communication between nodes is now allowed. Therefore, we expand the description of a decision-making system to also include the update schedule $T$ (see the example at the end of section 3.3) of the involved decision makers together with the prior and the posterior models. We denote this set of descriptions the communication structure of the decision-making process.

In this section, we illustrate the effects on the decision-making processes under more general communication structures and how these structures allow for various interpretations of repeated communication between the involved decision makers. In particular, in section 4.2, we distinguish three different information-processing protocols depending on whether the posterior choice rules are optimized for each time step or continually updated and on whether the priors are temporally fixed or adaptive. We also demonstrate how these protocols can be interpreted using our running example from the previous section. In section 5, we provide further examples.

As we drop the prerequisite of prior and posterior models to be represented by acyclic graphical models, the sets of leaves as well as the roots of the now arbitrary graphs $G$ might be empty. Additionally, as there no longer exists a viable topological sort for cyclic graphs, there is no longer a corresponding information processing schedule. Thus, for graphs that allow for repeated communication between decision makers it is necessary to define when and potentially how often decision makers communicate.

For example, if we change the posterior model, equation 3.12 of our running example, such that decision maker $X2$ has a posterior of the form $P(X2|X0,X4)$ instead of $P(X2|X0)$⁠, introducing a feedback loop from decision maker $X4$ to $X2$⁠, the update rule from the previous section is no longer applicable as it is not defined which of the decision makers $X2$⁠, $X3$⁠, and $X4$ should be updated first, how long their communication lasts, and which of the decision makers should have the final update.

To tackle this problem, we define a general update schedule $T={T1,⋯Tδ}$ to be a family of sets $Tt⊂XI$ that contain the nodes that update their prior to their posterior at step $t$ of the decision-making process. Note that each node of the decision-making process needs to appear at least once in the sets $Tt$⁠. The introduction of the general update schedule means that instead of a single decision maker, an arbitrary set of decision makers is allowed to process information at each step, and, analogously to how each posterior in equation 3.9 depends on the information-processing costs of all subsequent nodes (as part of the effective utility, equation 3.8), the posteriors of the nodes in the set $Tt$ shall depend only on the costs of the decision makers in the sets $Tt+1,...,Tδ$⁠.

The choice of the update schedule $T$ is problem specific and describes when during the course of the process the decision makers communicate, whereas the prior and posterior models describe with whom the decision makers communicate. We call the integer constant $δ$ the horizon of the decision-making process.

Analogous to the feedforward case, we find realizations of the decision-making process by Gibbs sampling. As the product of all posteriors of a loopy decision-making system no longer forms a joint distribution over all decision variables, we consider the empirical distribution resulting from Gibbs sampling as an approximation of the actual joint distribution of the variables instead.

So far, we have treated the posterior and prior updates inside the Blahut-Arimoto iteration of alternating equations 3.9 and 3.10 to remain the same as in section 3.3, just that now, we use the effective utilities, equation 4.2, for the posteriors of the nodes in $Tt$⁠. There are, however, more choices to be made considering the fact that in a loopy graph, each node is allowed to process information more than once. In particular, we consider priors to be either adaptive or fixed over communication steps, and moreover, we consider decision makers to either have a single posterior for all communication steps or to have a separate posterior for each step. Figure 4 displays the resulting information-processing protocols for our running example, which are described in more detail in the following section.

From the point of view of probability theory, we would treat each occurrence of $Xk$ in the schedule $T$ as a separate random variable with its own prior and posterior, attempting to transform the loopy graph back to a directed feedforward decision-making graph as studied in Gottwald and Braun (2019) and summarized in section 3. With such an approach, however, one would have to relabel nodes that appear multiple times and then decide which relabeled version should be used as input for which subsequent node in the unrolled graph. In larger graphs with multiple loops, there would be many such decisions, essentially replacing the loopy decision-making problem with an entirely different one. Instead, here, we keep the loopy dependency structure not only for the purpose of inferring the priors and effective utilities using loopy belief propagation, but also to motivate different processing protocols $P$⁠, depending on which information is shared among multiple instances of the same decision maker in $T$ (see algorithm 2 in the appendix for the resulting procedure).

In particular, we consider the following set of protocols $P$⁠:

• Protocol 1: Each node in a cycle is represented by a posterior and prior that is optimized for each update step. The optimal priors are determined in each cycle from belief propagation.

• Protocol 2: Same as protocol 1, except that the priors are never changed from their initial setting.

• Protocol 3: Each node is represented by a continuously updated posterior. The prior of each node is given by the previous posterior of that node.

Notice that since the update schedule $T$ is no longer grounded in a topological sort of the nodes of the graphical model, the same prior and posterior models can be used to describe multiple different decision-making processes that allow for repeated communication, whereas in the feedforward case, variables are considered to be updated according to the information flow (i.e., according to a corresponding topological sort of the nodes).

In the first protocol, we assume a separate posterior $Pkt$ together with its optimal prior $pt$ for each occurrence of decision maker $Xk$ in $T$⁠, related by equation 4.3. However, in contrast to unrolling over time, we do not rename the decision makers, but rather consider time-dependent probability distributions. In particular, we keep the loopy graph for belief propagation to infer the priors as the marginals of the corresponding posteriors and to infer the auxiliary distributions $pt(·|xk,xA(k))$ that determine $Ukt(xk,xA(k))$ according to equation 4.2.

For our running example, this means, for example, for decision maker $X4$⁠, who processes information at $n1=3$ and $n2=6$⁠, that beliefs of the priors $p3(X4|X3)$ and $p6(X4|X3)$ are determined using belief propagation, together with the auxiliary distributions $p3(X{0,2}|X{1,3,4})$ and $p6(X{0,2}|X{1,3,4})$ to determine $U^43$ and $U^46$⁠, which allows calculating $P43(X4|X{1,3})$ and $P46(X4|X{1,3})$ according to equation 4.3.

This update protocol fits a situation of repeated communication where continually adapting to the optimal prior is advantageous. A negotiation that consists of multiple rounds or a coordination game in which a player earns higher payoff whenever he chooses the same course of action as another player would be examples of such scenarios. In section 5 we provide a more detailed example of a bounded rational decision-making system that represents players in such a coordination game.

One could also imagine that the timescale of a communication process is too short for a decision maker to change its prior, which could be considered to change only in the long run. Hence, we might assume that the time dependency of the posteriors, equation 4.3, originates only from the effective utilities $Ukt$⁠. We have therefore the same expressions for $Pknj$ and $U^knj$ as in protocol 1, but with a fixed prior $pnj=p$⁠, which, for simplicity, in our experiments, is taken to be the belief of the marginal over the posteriors ${Pn1k}k=1N$⁠, where $n1k$ denotes the first occurrence of node $k$ in the update schedule $T$⁠, the first time decision maker $k$ processes information. However, one could also imagine a marginal over time, where all decisions in $T$ are averaged.

With respect to our running example, this would mean that decision maker $X4$ would have a fixed prior $p(X4|X3)$ that is used for both posteriors, $P3$ and $P6$⁠.

The previous two protocols can be considered to model a communication process, resulting in a set of posteriors ${Pkt}k,t$⁠, one posterior for each node $Xk$ and time $t$ (or rather for those $t$ for which $Xk∈Tt$⁠), from which we can sample according to the schedule $T$ to simulate an optimal process obtained by the Blahut-Arimoto iteration.

Coming back to our introductory example from the end of section 3.2, where a student (node $X1$ in Figure 1a) learns to react to a question asked by a teacher (node $X0$ in Figure 1a), information processing according to this update protocol can be thought of as a repeated process with repeated access to the utility function, where the posterior is changed step by step rather than in a single instant, for example, because the student is granted repeated access to interactions with a (for simplicity, constant) teacher, resulting in increased performance.

In this section, we present examples of decision-making systems with general communication structures and measure their performance in terms of their expected utility.

Continuing the introductory example of a student and teacher from section 3.2, we keep the simple graphical model that determines the prior and posterior models and modify the update schedule $T$ of the communication structure such that the student’s posterior is updated multiple times (see Figure 5) and compare the differences in performance. For this example, we choose to use protocol 3, because a posterior that is gradually updated fits the description of a student who improves performance by extending preparation time $δ$⁠. This way, a hard-working student (high $δ$⁠) with low computational resources (low $β$⁠) achieves similar expected utility as a student with high computational resources (high $β$⁠) but fewer repetitions (low $δ$⁠). Figure 5 illustrates this relationship. The trade-off between $β$ and $δ$⁠, or more generally between beta and the total number of updates to the decision makers in the decision-making process, is discussed in more detail in the following examples and the discussion in section 6.

We use the performance measured by the expected utility of the original feedforward decision maker averaged over all 1000 randomized utility functions as a baseline for the performance of the decision maker with the feedback loop and repeated communication. Figure 6 shows the baseline (red line parallel to the abscissa), the average performance of the loopy decision maker (blue dots) and a box plot containing the median, the first and third quartile of all performances for every value of $δ$ ranging from 2 to 20. All data are presented relative to the baseline.

As we can see, the relative performance level of the loopy graph increases with growing $δ$ and appears to be converging for higher values of $δ$⁠, implying that increased communication between intermediate and final node leads to a higher expected utility. This holds for the information processing protocols 1, 2, and 3. The plot in Figure 6 shows the results for the third protocol.

In game theory, a coordination game is a type of game in which a player will earn a higher payoff whenever they choose the same course of action as another player (Cooper et al., 1992; Cooper, 1999). Here, we consider a situation in which two players play one or more coordination games chosen from a set of game matrices (see Figure 7b). This situation is modeled by a decision-making system composed of an observation node $X0$ that decides on the type of game that is played and two decision nodes $X1$ and $X2$ that represent the players.

Considering the pure coordination game (game 1 in Figure 7b), a suitable game-theoretic solution concept is rationalizability (Bernheim, 1984), where it is assumed that both players act rationally and it is common knowledge that both players act rationally. In the pure coordination game example, player $X1$ plays up if he can reasonably believe that player $X2$ could play left because up is a best response to left. Additionally, $X1$ can reasonably believe that $X2$ plays left if $X2$ believes that $X1$ could play up. So, $X2$ believes that $X1$ plays up if it is reasonable for $X2$ to believe that $X1$ could play up. Continuing this argument generates an infinite chain of reasonable beliefs that leads to the players playing (up, left). Analogously, a similar process can be repeated for (down, right).

Here, we approximate this process with bounded rational decision makers for all the different games simultaneously by using a utility function that represents all of the game matrices (see Figure 7 for details). The decision makers are updated according to protocol 1 as all the deliberate decisions of the two player according to the above example are best represented by decision makers who find the optimal prior and posterior for every time step of the deliberation process.

Figure 7 summarizes the results of the simulation. Panels d and e show the expected utility achieved by decision-making systems for different combinations of $δ$ and $β$ parameters and update schedule. In this example, the schedule $Tone$ leads to better performance than $Ttwo$ across almost all combinations of $δ$ and $β$ parameters, evening out for high values of both $δ$ and $β$⁠. Furthermore, looking at the average information processing cost for both schedules and combinations of parameters (see Figures 7g and 7h), it is apparent that although the average processing cost for the update schedule $Ttwo$ is higher, its performance is worse. This means that, depending on the specific problem, fewer updates can lead to better results.

Analogous to previous studies (Genewein et al., 2015; Gottwald & Braun, 2019), the overarching principle behind the ideas presented in this article is the trade-off between an increase of a desirable quantity and a cost associated with a corresponding change in behavior. This trade-off is necessary as resources such as computational or cognitive capacities are typically limited and decision makers are forced to choose actions that are good enough rather than optimal. Here, we offer an additional approach to increase the expected utility of a decision maker. Like a student who repeats a difficult topic again and again or two partners in a cooperative game who argue back and forth multiple times, we allow the structure of the graphical model that describes the decision-making process to include cycles and feedback loops to mimic the gradual gain in performance of the student or the cooperative partners in each iteration of their respective information processing instance. Although graphical models with cycles open up these new pathways, they come with several drawbacks in the form of an increased challenge in inference tasks, an absence of convergence and correctness guarantees, a higher ambiguity in causal relationships, and the resulting lower level of interpretability and intuition behind the graphical model. To deal with some of these flaws, other authors have suggested methods that involved deleting edges in a directed graph (Castillo et al., 1998) or reversing edge orientations (Ariffin, 2018) to make them acyclic again. In contrast, we presented different information processing schemes to deal with the presence of loops in the graphical model of one or more bounded rational decision makers without deleting or changing any of the connections.

As outlined by Figures 6, 7d and 7e, and 5b to 5d, the number of updates $δ$ per Blahut-Arimoto iteration tends to contribute positively to the decision maker’s performance, and its influence resembles the influence of the $β$ parameter for every decision node in the decision-making process, often allowing to decrease the $β$ parameter (and correspondingly the actual boundary on the processing cost) while having the same or better performance for reasonably high values of $δ$⁠. Figure 5f captures this relation for the simple toy example showing that the necessary $β$ value for a specific performance threshold behaves antiproportional to the $δ$ parameter for this specific setting.

Although this seems to indicate that an increase in updates per node and per Blahut Arimoto iteration increases the overall performance of the decision-making system, increasing the number of updates is a double-edged sword as more updates lead to a larger impact of subsequent updates to the effective utility, which might result in a decrease in performance. Two examples for this are Figure 7e (in comparison to 7d) and Figure 8, where we increased the number of updates to the node by increasing the number of nodes in the terminal update list and varying the number of edges in a given graph, respectively. In both cases a larger average number of updates to every node leads to a decline in performance, which is loosely reminiscent of the problem of information overload (Schroder et al., 1967), which, in the context of this article, means that too many feedback connections decrease the overall performance of a system. Additionally, Figure 7h suggests that adding more nodes to the terminal update list $Tδ$ does not increase the overall performance although effectively doubling the number of updates to the nodes resulting in an increased information processing cost.

To summarize, there is no straightforward strategy to determine the best initiation of a graphical model for a given problem. Unlike the typically monotone increase in performance for growing $β$ parameters, the influence of the number of feedback loops and updates to the performance of different nodes in the graph stagnates and falls off for increasingly large values.

Nevertheless, feedback loops and recurrent processing have proven to be valuable tools, enabling top-down information processing and enhancing object recognition (Rao & Ballard, 1999; Ernst et al., 2021; Spoerer et al., 2020; Herzog et al., 2020). They also play a crucial role in modeling human-inspired attention mechanisms, including recent enhancements to transformer-like attention mechanisms with recurrent connections (Stollenga et al., 2014; Mittal et al., 2020; Zeng et al., 2021; Ju et al., 2022).

The examples provided in the previous section represent only a few applications where general communication structures consisting of a graphical model and an update schedule can be used. Although the flexibility of the unrestricted composition possibilities of nodes and edges to form graphical models and corresponding update schedules to describe communication for almost every scenario are one of the upsides of our approach, it makes a more exhaustive analysis (as it was done in Gottwald & Braun, 2019) of different communication structures intractable.

One disadvantage of the free energy from constraints approach is the choice of Lagrange multipliers $β$ that are introduced to the model. Their meaning comes from their connection to the constraints on the processing cost in the original constrained optimization problem, for which there is no closed form mapping that describes their specific relationship. This means that in order to satisfy a certain constraint, the values of $β$ have to be computed numerically, which can be computationally expensive.

Additionally, although we argue that loosening the restrictions and prior assumptions on the graphical model is possible, the decision-making process as presented in the article is still restricted to a single global utility function that is the driving factor behind the change of behavior of all decision makers involved in the decision-making process. This fact restricts the use of the bounded-rational decision-making models to a certain subset of problems. For example, in game theory, the decision makers can only deal with cooperative or coordination games, where all players receive the same reward for each decision realization. Noncooperative games would require individual preferences with the set of decision-making nodes; that is, the model would have to be modified to allow for local or individual utility functions alongside the global utility function that expresses the preferences of a subset of decision makers or a single decision maker. The incorporation of additional localized utility functions will be a subject for future research.

In the context of statistical modeling and inference, maximum (log-)likelihood estimation (MLE) is a popular approach to estimate model parameters by maximizing the likelihood of observed data. Maximum regularized likelihood estimation extends MLE by incorporating regularization terms into the objective function, which help to control the complexity of the model or impose desirable properties (Zhuang & Lederer, 2018). One popular regularizer is the Kullback-Leibler divergence, which discourages large differences between priors and posteriors (Le Cam, 1990; Ichiki, 2023). In the special case where the parameters themselves are modeled as random variables and, moreover, if the regularizer is the Kullback-Leibler divergence between its prior and posterior distributions, then one obtains (variational) Bayesian inference.

In this form, it is apparent that the choice for the utility $U(o,x):=βlogp(o|x)$ renders the variational free energy $Fvar$ to be identical to the free energy that is maximized in the one-step case from section 3. In particular, the bounded rational posterior $P(X|O)$⁠, in this case, coincides with the Bayes’ posterior $1Zp(X)elogp(O|X)=1Zp(X)p(O|X)$⁠.

Also, note that from a purely technical perspective, removing the $DKL$ term from both optimization objectives reduces the bounded rational decision-making problem to a rational decision-making problem (unconstrained utility maximization) and variational Bayesian inference to standard maximum log-likelihood estimation. Both the rational decision maker and the maximum log-likelihood estimator put all eggs in one basket, making them vulnerable to model misspecification. Thus, the $DKL$ term can simply be considered an entropy regularizer to utility maximization and log-likelihood estimation, respectively.

In summary, variational Bayesian inference has similar ingredients to information-theoretic bounded rationality: log-likelihood plays the role of the utility, and the entropy regularization that turns maximum log-likelihood into Bayesian inference plays the role of informational costs. As utilities can be arbitrary, whereas log-likelihoods are normalized such that they are probabilities when exponentiated, formally, our approach to systems of bounded rational decision-making units might therefore be considered as Bayesian inference using unnormalized log-likelihoods, allowing the description of networks of decision nodes that collaborate to optimize an arbitrary objective. Decision makers in the decision-making system that do not have a direct influence on the utility function serve as hidden signals that filter information for subsequent decision makers.

As outlined in section 2, message-passing algorithms are an invaluable tool in control as inference (Toussaint & Storkey, 2006; Kappen et al., 2012; Todorov, 2008; Toussaint, 2009; Levine, 2018), where actions are treated as state-like variables so that Bayesian inference can be applied for policy search.

In the literature on active inference and the free energy principle, however, message passing is lifted from a purely computational tool to serve as explanation for certain processes in the human brain (Friston et al., 2017). There, variational inference over actions (and other unknown random varibles) is considered under various approximations on the state and action distributions, ranging from rather restrictive assumptions such as the mean-field approach, which assumes statistical independence of all involved variables (Friston, 2009, 2010; Friston et al., 2017), to more capable assumptions such as the Bethe approximation (Schwöbel et al., 2018), which allows for pairwise statistical dependencies. The update equations resulting from optimizing the variational free energies under these approximations are then compared to message-passing equations and discussed in regard of their biological plausibility in terms of explaining inference processes in the brain.

In contrast, our use of message passing is in line with control as inference approaches, namely, as an efficient tool to perform approximate Bayesian inference in graphical models with loops. However, strictly speaking, we use belief propagation only to infer auxiliary distributions required for the effective utilities and priors that determine the action posteriors. Those are assumed to take the form of Boltzmann distributions and are therefore not determined directly using belief propagation (see sections 3 and 4). However, under a more general point of view, one could simply consider the Boltzmann distributions as a specific way to combine incoming messages from the surrounding nodes and factors, analogous to the product of incoming messages in the sum-product algorithm.

In this article, we have studied systems of bounded rational decision makers and extended the previous work of Genewein et al. (2015) and Gottwald and Braun (2019) by developing a generalization of the update algorithm that previously restricted the set of possible graphical structures to feedforward architectures, where nodes could only obtain information about other nodes if there was a directed path between them. Here, we argued that we can loosen those restrictions and expand the original algorithm to include arbitrary graphs and update schedules to form general communication structures.

To this end, we combined an inference tool, belief propagation, with bounded rational decision making and showed that increased levels of communication between decision makers can be a substitute for computational capacity whenever decision-makers are limited.

A central task in a multitude of inference problems is to calculate a marginal distribution $p(XS=xS)=∑xI∖Sp(xI)$ of a given joint probability distribution $p(XI)$ where $S⊂I$⁠. The sum-product message-passing algorithm is an efficient way to solve this task as it utilizes a (known) factorization of the joint probability. The algorithm is correct whenever the underlying graphical model is a tree and gives good approximate results for graphs that contain cycles (Murphy et al., 2013; Yedidia et al., 2003, 2005).