Abstract

We describe the construction and theoretical analysis of a framework derived from canonical neurophysiological principles that model the competing dynamics of incident signals into nodes along directed edges in a network. The framework describes the dynamics between the offset in the latencies of propagating signals, which reflect the geometry of the edges and conduction velocities, and the internal refractory dynamics and processing times of the downstream node receiving the signals. This framework naturally extends to the construction of a perceptron model that takes into account such dynamic geometric considerations. We first describe the model in detail, culminating with the model of a geometric dynamic perceptron. We then derive upper and lower bounds for a notion of optimal efficient signaling between vertex pairs based on the structure of the framework. Efficient signaling in the context of the framework we develop here means that there needs to be a temporal match between the arrival time of the signals relative to how quickly nodes can internally process signals. These bounds reflect numerical constraints on the compensation of the timing of signaling events of upstream nodes attempting to activate downstream nodes they connect into that preserve this notion of efficiency. When a mismatch between signal arrival times and the internal states of activated nodes occurs, it can cause a breakdown in the signaling dynamics of the network. In contrast to essentially all of the current state of the art in machine learning, this work provides a theoretical foundation for machine learning and intelligence architectures based on the timing of node activations and their abilities to respond rather than necessary changes in synaptic weights. At the same time, the theoretical ideas we developed are guiding the discovery of experimentally testable new structure-function principles in the biological brain.

1  Introduction

We describe the construction and theoretical analysis of a framework that models how local interactions among connected node pairs can be used to compute the global dynamics of a class of spatial-temporal network. By analyzing the local dynamics of incident signals into nodes and how signals compete to activate downstream nodes, we were able to prove a number of properties associated with the dynamics of the network. This framework was derived from a theoretical abstraction of the canonical neurophysiological principles of spatial and temporal summation in biological neurons (Magee, 2000; Kandel, Schwartz, Jessell, Siegelbaum, & Hudspeth, 2012). We considered spatial-temporal networks in the sense that we analyzed signals along (geometric) directed edges in a metric space that give rise to temporal latencies or delays of signals. The metric space is not intended to conform to any specific coordinate system. The only requirement is that it be able to support a spatial embedding that preserves geometric distances between edges, because it is the geometry of the edges that gives rise to the signaling latencies given a conduction velocity (signaling speed). When combined with a refractory state for each node, a central concept in this model, the interaction between the two has a dominating effect on the global network dynamics. Following successful activation, every node experiences a subsequent period, or state, of refractoriness, during which it is incapable of responding to subsequent inputs. The framework we developed reflects local processes from which global behaviors emerge.

At its core, our model takes into account how the timing of different signals influences their competition to activate target nodes they connect into, given that the internal (refractory) state of the node they are competing for may or may not allow that node to be activated. The refractory period is a value that reflects the internal state of the node in response to its activation by an upstream winning node (or set of nodes). In contrast to most other work, we do not assume anything about the internal model that produces this refractory state. For example, this could include an internal processing time during which the node is making a decision about how to react or an internal reset period after the node has reacted before it is in a state capable of responding to new inputs. A refractory period is a reasonable assumption for any physically constructible network, since infinitely fast processing times are not realizable. If the timescale of such a refractory state is much shorter than the time it takes for propagating signals to reach their vertices, due to either fast signaling speeds or short edges, it can have a significant affect on network dynamics and function. The inverse will also affect the dynamics, where the refractory period is much longer than the time it takes for signals to arrive at target nodes. These concepts are foundational to the analysis we present in this work.

In order to intuitively capture the relationship between signal flows on edges and the refractory states of the participating nodes, we defined a refraction ratio for each node. For every pair of connected nodes in the network, we define this as the ratio between the amount of time left in the recovery from the refractory period for the node receiving a signal, relative to the amount of time before the next signaling event arrives at that node from the upstream node that connects into it. The refraction ratio captures the relationship between the speed and temporal delays of signaling or information flow, which is bounded by the spatial geometry of the edges, and the refractory state of the node. We use the term activation here to imply an appropriate reaction of that node to the signal it has received. In the case of biological neurons, for example, it would be the generation of an action potential that propagates down the axon and axonal arborizations to synaptic terminals. In neurons, the membrane refractory period due to the biophysics of the membrane ensures the directional nature of the traveling action potential wave along the axon and sets the cell's maximal firing frequency. Analogously in our framework, it is the generation of discrete signaling events that directionally travel between edges that connect the nodes in a network. As we and others have previously shown, the interplay between temporal latencies of propagating discrete signaling events, relative to the internal dynamics of the individual nodes, can have profound effects on the dynamics of a network (Buibas & Silva, 2011; Manor, Koch, & Segev, 1991; George & Kim, 2013). In a spatial-temporal network, as we define it in this work, temporal latencies need not be directly assigned but result from the relationship between signaling speeds (conduction velocities) and the geometry of the edges on the network (i.e., edge path lengths).

An analysis of the framework led to two main results. First, a series of systematic extensions of the basic construction resulted in a spatial-temporal version of a perceptron. In the classical perceptron, contributing weight summations from upstream nodes into an activation function determine whether that node produces an output signal (Rosenblatt, 1958; Minsky, 1967). In our case, we considered how temporal latencies produce offsets in the timing of the summations of incoming discrete signaling events. Signaling latencies affect when and how the threshold of the perceptron's activation function is reached as a function of a running summation in time. This produces a much richer dynamical repertoire relative to the classical perceptron model. As part of this model, we defined a decaying memory function such that the maximum value of the weight for a given connection occurs at the time of arrival of the signal. Subsequent time steps result in progressive decreasing (i.e., decaying) contributions from the maximum weight value to the summation function.

Second, a theoretical result from the analysis of the refraction ratio was a set of bounds that allowed us to formally define a notion of efficient signaling within the context of the framework. We show that an optimal ratio is one where the timing of information propagation between connected nodes does not exceed the internal dynamic timescale of the nodes. In other words, it represents a balance between how fast signals propagate through the network relative to the time needed for each node to process incoming signals. Efficient signaling in the context of the framework we develop here means that there needs to be a temporal match between the arrival time of the signals relative to how quickly nodes can internally process signals. A mismatch between these two considerations can cause a breakdown in the signaling dynamics of the network. We have previously shown in numerical experiments that a qualitative imbalance of this phenomenon can result in the breakdown of a network to sustain recurrent activity (Buibas & Silva, 2011). Intuitively, efficient signaling as we develop it in this work implies that there is necessarily a temporal match between the amount of time it takes signals to travel between connected node pairs in a network relative to how quickly the nodes receiving signals can internally process them. A mismatch in time between these two considerations results in a breakdown in the signaling dynamics of the network. It is this notion of efficient signaling that we formalize here.

The focus of our exposition in this letter is the development and theoretical analysis of the framework, not a numerical or computational investigation. We also intentionally do not show any applications of the work, which are beyond the intent of this letter. Ongoing work is using the framework to develop a new machine learning architecture. This architecture is fundamentally different from existing artificial neural network models in how it learns and encodes information and data. We are also using the refraction ratio in the neurophysiological analyses of structure-function dynamics in biological neurons and networks. Using high-resolution morphological reconstructions of axon shapes, we recently showed that the refraction ratio reflects a design principle that biological neurons optimize. It reflects a balance between the wiring lengths of axons (material cellular costs) versus signaling speeds (temporal costs; Puppo, George, & Silva, 2018).

The letter is organized as follows. In section 2 we introduce the basic construction of the competitive refractory framework and provide an analysis of its dynamics. Following a set of preliminary definitions in sections 2.1 and 2.2, we introduce the refraction ratio in section 2.3. We then show in section 2.4 how we can use this ratio to compute the set and order of winning nodes in parallel across a network, essentially providing an algorithm to compute global behaviors from local dynamics at the scale of individual node pair interactions. Section 3 outlines a number of systematic extensions of the basic construction. In section 3.1, we discuss a simple probabilistic extension of the deterministic version of the framework. Section 3.2 introduces inhibitory inputs into a node (in contrast to excitatory inputs). In section 3.3, we discuss the role and contribution of internal node processing times to subsequent signaling latencies on edges. Section 3.4 concludes with the development of fractional node contributions summating in time in order to reach an activation threshold that activates a downstream node. This is our model of a geometric dynamic perceptron. Section 4 introduces two distinct versions of graphics derived from Feynman diagrams intended to provide a visualization tool for the dynamics. In section 5, we introduce a notion of optimal efficient signaling within the context of the framework. We define a set of bounds the refraction ratio must conform to. Section 6 provides some concluding comments.

2  Competitive Refractory Framework

We considered the geometrical construction of a spatial-temporal network in the following sense. We assume that signals or discrete information events propagate between nodes along directed edges at a finite speed or conduction velocity, resulting in a temporal delay or latency of the signal arriving at the downstream node. Imposing the existence of signaling latencies implies a network that can be mapped to a geometric construction, where individual nodes could be assigned a spatial position in space in R3 for an ordered triplet x¯i=(x1,x2,x3) for each vertex vi for all vertices i=1,,N, where N is the number of nodes and therefore the size of the network. We use the terms vertices and nodes interchangeably throughout the letter. Formally, vertices belong to a graph that models a particular network, with the nodes belonging to the network. Directed edges connecting node pairs could have a convoluted path, that is, a Jordan arc. There is no restriction that edges have to be spatially minimizing straight-line edges (see Figure 1). A signaling latency τij expresses the ratio between the distance traveled on the edge, dij=|eij|, relative to the speed of the propagating signal sij, between a vertex vi that connects into a vertex vj. For all pairs of connected vertices vivj, τij=dij/sij. While one does not have to explicitly consider dij and sij, the existence of signaling latencies can always be mapped to these variables. This is analogous to the conduction velocity of action potentials traveling down the convoluted axon and axonal arborizations of a biological neuron (see Figure 1B).

Figure 1:

(A) Spatial-temporal geometric graph models of a network as we define them here are graphs whose vertices can have a physical position in space (or the plane), with directed edges that have a physical (geometric) representation in space (or the plane). Given a finite speed for discrete signals propagating down an edge, this produces signaling latencies or delays that when combined with a node (vertex) refractory state results in the dynamics of the framework. In constrast, in a classical representation of a graph, the only important consideration is the connectivity of the vertices (i.e., adjacency matrix). There is no geometric signficance to how the graph is drawn. See the text. (B) An example of a real-world spatial-temporal geometric network. Biological neurons can be modeled as tree graphs with the initial node at the cell body (soma) where the axon begins. The axon and its arborizations have convoluted morphologies (edge paths) and display discrete signals that propagate at finite speeds (action potentials). They are physically not edge path length minimized. (Panel b is adapted from Puppo et al., 2018.)

Figure 1:

(A) Spatial-temporal geometric graph models of a network as we define them here are graphs whose vertices can have a physical position in space (or the plane), with directed edges that have a physical (geometric) representation in space (or the plane). Given a finite speed for discrete signals propagating down an edge, this produces signaling latencies or delays that when combined with a node (vertex) refractory state results in the dynamics of the framework. In constrast, in a classical representation of a graph, the only important consideration is the connectivity of the vertices (i.e., adjacency matrix). There is no geometric signficance to how the graph is drawn. See the text. (B) An example of a real-world spatial-temporal geometric network. Biological neurons can be modeled as tree graphs with the initial node at the cell body (soma) where the axon begins. The axon and its arborizations have convoluted morphologies (edge paths) and display discrete signals that propagate at finite speeds (action potentials). They are physically not edge path length minimized. (Panel b is adapted from Puppo et al., 2018.)

2.1  Preliminaries

The set of all edges in the graph G=(V,E) is given by E={eij}, where eij denotes the directed edge from vertex vi to vertex vj. We define the subgraph Hj as the (inverted) tree graph that consists of all vertices vi with directed edges into vj. We write Hj(vi) to represent the set of all vertices vi in Hj and Hj[vi] to refer to a specific viHj(vi). We assume there exist discrete signaling events traveling at a finite speed sij on the edge eij. The signaling speed sij from vi to vj is bounded such that 0<sij<; that is, it must be finite. We introduce the notation Hj[vi]vj to mean a vertex viHj(vi) that causally leads to the activation of vj. In other words, this notation indicates the “winning” vertex whose signal manages to activate vj (see Figure 2).

Figure 2:

The subgraph Hj(vi) in relation to an illustrative arbritrary network. Each vertex vV has its own Hj subgraph. Note the relative use of the indices i and j. Vertex vj becomes one of several vi vertices that in turn connects into other Hj subgraphs. See the text for details.

Figure 2:

The subgraph Hj(vi) in relation to an illustrative arbritrary network. Each vertex vV has its own Hj subgraph. Note the relative use of the indices i and j. Vertex vj becomes one of several vi vertices that in turn connects into other Hj subgraphs. See the text for details.

We then define an absolute refractory period for the vertex vj by Rj. This reflects the internal dynamics of vj once a signaling event activates it—for example, the amount of time the internal dynamics of vj requires to make a decision about an output in response to being activated or some reset period during which it cannot respond to subsequent arriving input signals. We place no restrictions on the internal processes that contribute toward Rj. We assume that we do not know and cannot observe the internal model of vj, which could be quite complex (e.g., a family of temporal differential equations that produce its refractory period). But we do assume we can observe or measure how long Rj is. We also assume that Rj>0, that is, there cannot exist an infinitely fast or instantaneous recovery time, even though it can be arbitrarily short. This is a reasonable assumption for any physically constructible network.

Consider a vertex vi with a directed edge eij to a vertex vj. For vi to signal vj, there must be some discrete physical signal representing a flow of information from vi to vj over the edge that connects them. This signal must travel at some finite speed sij. sij could be a constant value for all edges, but this need not be true in the general case. Similarly, if all nodes vj in a network share the same internal dynamics, then Rj=RvV. But the framework does not assume this and can accommodate differing node-specific values of the refractory period. Once vj receives a signal from vi it becomes refractory for a period Rj and will not be able to respond to another incoming signal during this period of time. The temporal nature of Rj implies that as time progresses, it shortens and eventually decays to zero, at which time vj is able to respond to another input.

2.2  Internal Dynamics of vj

Let yj(Ωj,t) represent the instantaneous state of vertex j as a function of time and some (possibly unobservable) model or activation function with variable and parameter set Ωj. The internal state can be interpreted as a binary function determined by yj(Ωj,t). We can define this function at some observation time To as
yj(Ωj,To)=1,iffvjcanrespondtoaninput0,iffitisrefractorytoanyinput.
(2.1)

Once the winning vertex “activates” vj, it becomes refractory for a period of time Rj during which yj=0, determined by its internal dynamic model. An important point is that if the state of vj at some arbitrary observation time To is yj=0, it could remain refractory for some time <Rj if it had become refractory prior to To. This situation is interesting because we have to take into account phase shifts in τij and Rj at To in order to understand the patterns of node activations—in other words, which arriving signal results in the activation of vj is dependent on the amount of the refractory period of vj remaining relative to the time of arrival of the signal. We call the amount of residual refractory period at To the effective refractory period, which we write as R¯j. This is at the core of our model.

Algorithmically, we can compute at discrete times in parallel for every vertex in the network (i.e., every vj), which viH(i) causally activates vj (i.e., Hj[vi]vj). We will achieve this by keeping track of the temporal interplay of propagating signaling events on the edges relative to the refractory states of the individual vertices by computing the refraction ratio for all node pairs.

2.3  The Refraction Ratio

We first establish a simple and intuitive relationship between Rj and τij for a vertex vj. We define the refraction ratio between the refractory period Rj and a signaling latency τij associated with a discrete signaling event coming from a vertex vi on the edge eij as
Δij=Rjτij=Rj·sijdij.
(2.2)
Our determination of Hj[vi]vj will emerge from an analysis of this ratio for every vertex connected into vj in Hj(vi).

Before proceeding further, we point out a number of trivial and unallowable conditions that are necessitated by the physical construction of real-world networks and the definitions above. The trivial lower bound occurs as Rj0, yj=1 at all times. But recall that Rj>0. Rj=0 implies a nonrefractory vertex capable of instantaneous recovery to an incoming signal from an upstream vertex, a condition that is not physically realizable. The trivial upper bound occurs as Rj, yj=0 at all times, in which case there would be no information flow or signaling ever. As τij0, Δij becomes undefined, which is equivalent to stating dij0 since τijdij for a fixed signaling speed sij. Equivalently, τij0 if sij=. But these conditions are unattainable. Δij is necessarily restricted to finite dynamic signaling and information flow in a network.

Intuitively, for any vi into vj when yj=1, if every viHj(vi) initiates a signal at exactly the same moment and vj is not refractory, the vertex with the shortest edge path will win and activate vj. In other words, assuming a constant signaling speed sij for all viHj(vi) if we let Dij:={dij=|eij|:i=1,2,N} be the set of all edge paths for Hj(E), then Hj[vi]vj will be achieved by vi(minidij) for dijDij.

Under realistic conditions, at an arbitrary observation time To, there is likely to be a temporal offset between when each vi signals and how far along vj is in its recovery from its refractory period due to a previous signaling event. Furthermore, each viHj(vi) is statistically independent from every other vi, so that the amount of temporal offset for each vi vertex signaling vj will be different. In order to compute these offsets and keep track of the overall dynamics of the network, we need to index two different notions of time. We define ti to be the moment at which vi initiates a signal along its edge eij toward vj. The other is the observation time To itself, which is the moment at which we observe or measure the state of Hj(vi). Put another way, it is the moment in time at which we chose to interrogate how far along each signaling event is on its respective edge eij. Such an offset in the progression of discrete signaling events along different edges is the general case, except in the special case when every viHj(vi) signals at the same moment.

With these considerations in mind, we can take into account temporal offsets by slightly expanding how we define the refractory period and signaling latency. For the refractory period, let φj represent a temporal offset from Rj, such that at To,
R¯j=Rj-φjwhere0φjRj.
(2.3)
We call R¯j the effective refractory period. It reflects the amount of time remaining in the recovery from Rj at observation time To. When φj=0, it implies vj became refractory exactly at To. When φj=Rj, it implies that vj is not refractory and can respond to an input from any vi at any time. Note how when φj=Rj, vj may have been refractory at some time tTo-Rj, but assures the condition that R¯j=0 at To. For values 0<φj<Rj, R¯j is partially recovered. In other words, vj had become refractory before To but has not yet fully recovered at To. Note that unlike Rj, which is an actual variable of the dynamical properties of the network, φj and R¯j=0 are explicitly dependent on the relative observation time To. One could define them in absolute time as per the discussion in section 2.5. However, as we will argue, there are some practical advantages and even necessary conditions to consider the state of the network relative to an observation time To (see section 2.5).
In the most general case, the times ti for all viHj (i.e., the times at which each vertex initiates a signaling event relative to the observation time To) would not be expected to be all the same. At To, a signal from any vi may be traveling partway along eij at a speed sij, effectively shortening τij. Or it may be delayed in signaling if vi signals some time after To, effectively lengthening τij. Each vertex viHj(vi) is independent of every other vertex in the subgraph Hj(vi). As such, we expect different temporal offsets due to how far along a discrete signaling event is in its propagation along their respective edge eij. As such, we need to take into account the distance that a discrete signal initiated by each vi has traveled along its edge at the observation time To. To accomplish this, we extend how we consider a signaling latency in the following way. First, we retain τij to represent the absolute temporal delay (latency period) for a signal that travels on the edge eij for a vertex viHj when vi initiates a signaling event at ti. Note that the initiation of the signaling event could come before, right at, or after the observation time. We then define a temporal offset for τij, an effective shortening or lengthening of τij relative to To as follows,
τ¯ij=τij+δij,
(2.4)
where, δijR, δij>0 represents an effective delay or elongation beyond τij. In other words, it represents the vertex vi initiating a signal at some time after To. Values -τij<δij<0 represent an effective shortening of τij. This would be the case when vi had initiated a signal that was traveling partway along the edge eij toward vj prior to To. Using a consistent notation, we can write this remaining portion of the edge length as |e¯ij|. When δij=0, it implies that vi signals exactly at the moment the network is observed. And when δij=-τij, it implies that the signal arrives at vj at the moment the network is observed. Values of δij<τij, which result in τ¯ij<0, represent a signal arriving at vj prior to the observation time To.
We can now extend equation 2.2 to reflect the effective refractory period and effective latency (relative to an observation time To) as
Λij=R¯jτ¯ij.
(2.5)

The extra degrees of freedom that result from the temporal offsets in the timing of arriving signaling events at vj for the set viHj(vi) relative to each other produce a much larger combinatorial solution space for how vj may be activated, although we do not systematically fully investigate the size of this space in this letter. The global dynamics of the network results from the local statistically independent dynamics of each vertex and its corresponding Hj(vi). Once vj is activated, it contributes to the dynamics of the activation of the downstream vertices it connects into. The activation of vj results in its contribution as one of the viHj(vi) for each of the Hj(vi) it is a part of.

2.4  Analysis of the Refraction Ratio

Intuitively, the “winning” vertex viHj(vi) that successfully achieves the activation of vj, Hj[vi]vj, will be the first signaling event that arrives at vj immediately after vj has stopped being refractory. This is equivalent to stating that Hj[vi]vj will be achieved by the vi with the smallest value of τ¯ij larger than R¯j. This condition guarantees Hj[vi]vj. This will be the case for the value of τ¯ij such that τijRj+ (i.e., approaches Rj from the right)—that is, τij is slightly longer than Rj. The analysis of Hj(vi) will therefore necessitate computing the order of arriving signaling events to determine which one meets this condition first. This is effectively an algorithm that computes Hj[vi]vj at To.

We begin by defining a well-ordered set of refraction ratios for which the condition τijRj+ is met. First, define the set
Λijo:={Λij:i=1,2,N|Λijforτ¯ijR¯j+}.
(2.6)
This implies that every ΛijΛijo satisfies the condition Λij<1. This is nothing more than a consequence of how we defined the ratio in equation 2.5, with the refractory period in the numerator and the latency in the denominator. Next, we impose an additional structure on Λijo by ordering it with the standard > operator. We order the set from the largest refraction ratio to the smallest. Note that although we do not make explicit use of the ordering of Λij yet, we will in section 3.4. Because Hj is finite, it is a subgraph consisting of a finite number of vi vertices that connect into vj; therefore, Λijo will be a finite set. We can then use Λijo to compute viHj[vi]vj using the following theorems. Doing so for all Hj(vi) in the network in parallel at measurement times To allows us to compute the global state and dynamics.
Theorem 1.
Assume vj has an effective refractory period R¯j at an observation time To. If φjRj, then the condition Hj[vi]vj is given by the refraction ratio ΛijΛijo that satisfies
vi=max(Λij).
(2.7)
If φj=Rj, then the condition Hj[vi]vj is given by
min(τ¯ij)viHj(vi).
(2.8)
Proof.

Let the set Λijo be defined as in equation 2.6. When φjRj, it guarantees that R¯j0. This means that the effective refractory period of vj is either partially recovered from the absolute refractory period at the observation time To if 0<φj<Rj, or else Rj¯=Rj when φj=0 (see equation 2.3). If vj is still refractory for some period R¯j, the signal with the smallest τ¯ij may not be the signal that activates vj because it may arrive before R¯j ends. In this case, the signal that arrives first immediately after R¯j ends will satisfy Hj[vi]vj. Because of how the refraction ratio and the set Λij are defined, with ΛijΛijo<1 (see equation 2.5), Hj[vi]vj will be met by the vertex vi that satisfies max(Λij), where the notation · indicates the ceiling function, as is typical. To see this, assume some constant value R¯j=a>0. Let τ¯(I+1)j>τ¯Ij for two τ¯(I+1)j and τ¯Ij with ΛijΛijo. Then a/τ¯(I+1)j>a/τ¯Ij. The closest of the two values of τ¯ij to 'a' then is τ¯Ij. For a set Λijo=a/τ¯Ij>a/τ¯(I+1)j>a/τ¯ija/τ¯Nj, the condition τ¯ijR¯j+ first relative to all ΛijΛijo is the largest ratio closest to unity, that is, max(Λij).

If φj=Rj, then R¯j=0Λij=0, and the “winning” vertex Hj[vi]vj will be given by min(τ¯ij)viHj(vi), since Λij=0 only when R¯j=0 and y¯j(Ωj,t)=1. In other words, if vj is not refractory at To then the first signaling event that reaches it is guaranteed to activate vj. This will be given by the trivial condition of the signal with the smallest effective latency τ¯ij.

Alternatively, we can equivalently express theorem 1 as follows:

Theorem 2.
Assume vj has an effective refractory period R¯j at an observation time To. Then for each viHj(vi) with associated τ¯ij, the condition Hj[vi]vj will be satisfied by
min[(τ¯ij-R¯j)>0].
(2.9)
Proof.

The necessary condition for Hj[vi]vj is the vertex with a signal that satisfies τ¯ijR¯j+ first (see the proof of theorem 1). Given pairs of value (R¯j,τ¯ij) for each ΛijΛijo and a constant value R¯j at To, max(Λij) must also satisfy min[(τ¯ij-R¯j)>0], given that the ratio of vi has max(Λij).

Algorithmically, equation 2.9 is more efficient to implement because one only needs to compute a difference compared to equation 2.7 which necessitates computing a ratio. This becomes significant when computing all HjG(V,E) in parallel.

2.5  Dynamics in an Absolute versus Relative Temporal Reference Frame

In a strict sense, the criterion for successful activation of a node vj can be described using a continuous absolute time frame of reference without the need for considering an observation time To. This can be achieved in a straightforward way. Let Rj0 be the start of the refractory period Rj at the moment vj is activated by Hj[vi]vj. The necessary condition for activation τ¯ijR¯j+ can then be restated as Rj0+Rj<ti+τij. Rj0+Rj reflects the time at which vj stops being refractory. If the node becomes refractory at Rj0 and its refractory period lasts for a period Rj, then it will stop being refractory after Rj0+Rj. The condition for activation will be met when the signal from vi arrives after this time—namely, after ti+τij. Correspondence with the description above that makes use of a relative framework of observation time can be recovered by the relationship Rj0+Rj-To<ti+τij-To.

While mathematically accurate, this approach has a number of important functional and practical limitations that necessitate the consideration of the refraction ratio not from the perspective of absolute continuous time but from a relative observation or measurement time. A formulation of the model in terms of the absolute times ti and Rj0 requires that one be able to observe the network from the start of the evolution of the dynamics—in other words, from a dynamically quiescent state. This is because one has to be able to observe the various times ti and Rj0 for all the nodes in the network as they happen in order to compute the corresponding refraction ratios. A problem arises, however, if one needs to calculate refraction ratios for a network that has not been observed from the start of its dynamics. In this case, past times ti and Rj0 from the moment the dynamics starts are unknowable. This is indeed the case for networks that never turn off (e.g., the brain) or in the context of machine learning frameworks that make use of this model whereby the machine is initiated and allowed to run for a period of time prior to its observation—for example, at the time when data for learning are presented to it and the network computationally interrogated. Similarly, exploring the refraction ratio of neurobiological preparations makes it impossible to know the entire dynamical history of the system. While making appropriate measurements at some To may be difficult depending on the system being studied, one is at least guaranteed that computation of the winning vertices and evolution of the dynamics can in principle be computed and avoids a condition that would guarantee an inability to track the system's dynamical history—namely, needing to know unobserved measurements from the past.

A related consideration is that even if one were able to observe the dynamics from its inception (i.e., the total history of the system), it could be very computationally expensive to do so. But by referencing a relative observation or measurement time To, the same calculations are based only on the current (observable) state of the dynamics, not on its past. This could save significant memory resources, since the amount of information about the system that needs to be retained is much much smaller.

3  Extensions Beyond the Basic Construction

In this section we introduce a number of natural extensions to the basic construction of the framework. We first discuss a probabilistic version of the framework, an approach for dealing with inhibitory inputs, and how the explicit contribution of the internal processing time of nodes affects the timing of subsequent output signals, which in turn affects the refraction ratio. We conclude by considering a version of the framework that accounts for a fractional contribution of summating signals. This in effect represents a geometric dynamic version of the classical perceptron. These are not mutually exclusive, and one can consider an implementation of the framework that simultaneously accommodates any or all of them, adding to the dynamical richness of the model.

3.1  Probabalistic Extension of the Framework

The construction of the framework as introduced so far is deterministic in that one individual vi node is capable of activating vj, Hj[vi]vj. In other words, an activation of vj by vi is guaranteed to produce an output. This is equivalent to stating that the probability of vj responding to a “winning” signal from vi is one. However, we can extend these concepts to add a probabilistic element to the framework. For example, biological neural networks are not deterministically precise and are often subject to random fluctuations in how one neuron signals another due to thermal dynamical and subdiffusion considerations associated with molecular events such as neurotransmitter vesicles crossing the synaptic cleft or the stochastic nature of binding events on the postsynaptic membrane. Such a probabilistic extension would more accurately capture these aspects of the neurophysiology. An open question, however, is to what degree such probabilistic considerations improve the functional properties of the model in an engineering sense.

We assign a probability distribution Pij to the likelihood of activation of vj for a winning vertex Hj[vi]vj. vj(Pij) indicates the output probability of vj for some threshold Pthreshold given that the condition Hj[vi]vj has occurred:
vj(Pij)=1,ifHj[vi]vjandvi(Pij)Pthreshold0,ifHj[vi]vjandvi(Pij)<Pthreshold.
(3.1)
It is important to realize that equation 3.1 does not specify what the output from vj will be given vj(Pij)=1, which could be an actual signal if vi is excitatory or no signal if it is inhibitory (see section 3.2), only that vj does respond in some way to vi.

3.2  Inhibitory Inputs into vj

We can also extend the framework to include inhibitory inputs from vi in the following way. Given a “winning” vertex vi, Hj[vi]vj, vj generates an output signal in turn if the input from vi was excitatory or does not produce an output but still becomes refractory if the input from vi was inhibitory. The functional result is the lack of an output from vj (i.e., no contribution to the activation of its downstream vertices), while preventing further activation of vj until R¯j ends. This is not the only way one can think of differentiating between excitatory and inhibitory inputs, but it reflects a simple approach.

3.3  Explicit Contribution of the Internal Processing Time of vi

In some (most) cases, the activation of a node would not result in an instantaneous output but would require a finite period of processing time associated with the node's internal dynamics prior to the generation and output of a signal. This is a function of the internal dynamic model of the node independent of the network dynamics. The details of such an internal model can be node or node class specific. We do not need to make any such distinction or have any knowledge about what the internal model is. The consequence of an internal processing time on the dynamics of the network through an analysis of our framework is a contribution to the effective latency of the signaling event reaching downstream nodes. The longer the internal processing time is, the longer the effective latency will be. This is strictly different and independent from (that is, and can evolve in parallel with) the node's effective refractory period. If this internal processing time approaches the timescale of the signaling speed and refractory period, it will affect the dynamics of the network. If, however, it is much smaller than both, it will have a negligible effect on the dynamics and can be ignored. In the limit as it approaches zero, it reflects a (near) instantaneous turn around time between when a signal that activates the node arrives and when that node outputs a signal in turn.

Before continuing, we note a potential source of confusion with regard to the node subscript notation that has to be made clear. An internal processing time is a property of node vj following its activation. However, it affects the latency of an outgoing signal from vj; traveling along the edges, it connects to downstream nodes. In the context of signals leaving vj, its subscript in effect changes from j to i because it is now an input into the nodes it connects into. The subscript notation is of course relative to the context of the individual vi, vj node pairs under consideration. This is the mathematical equivalent of the relative usage of the terms presynaptic and postsynaptic when considering biological neuronal signaling. (A postsynaptic neuron receiving postsynaptic potentials from its presynaptic neurons becomes the presynaptic neuron once its own action potentials reach its synaptic terminals and the cell passes along the signal to the neurons it then connects to.) We attempt to be as clear and explicit as possible regarding this distinction in the description that follows.

Similar to the notation introduced previously, let tj be the time at which an activated node vj outputs a signal. Let tjin be the period of the internal dynamic processing time by vj. This is the time between when a “winning” signal Hj[vi]vj arrives at vj and when vj actually sends out an output at tj.

In the description of the framework in section 2, we implicitly assumed that tjin=0, reflecting an instantaneous output at the moment that vj is activated. Rj must begin anywhere between the moment of activation of vj when τ¯ij for the winning node ends, and tj. But this is strictly a property of the internal dynamics of vj. If tjin>0 (i.e., if there is some period of internal processing time between when vj is activated and when it sends an output signal at time tj), there will be a lag before vj initiates an outgoing signal. It affects the timing of when its signaling events reach the vertices it connects to. Recall that for any vertex vi, we denote the time at which it initiates an outgoing signal as ti. This is where the switch in index notation must occur. For an activated vertex vj with tjin>0, we write here its signal initiating time as ti' to indicate that this particular ti corresponds to the activated node vj.

From a computational perspective, we can absorb the effect of tjin>0 by an appropriate elongation in δij. Note that at this point, vi in the δij and τij subscripts in equation 3.2 correspond to the original vj that became activated, thus requiring a switch in index from i to j; the j index now refers to the downstream vertices that the original vj vertex is directionally connected to and signaling. We can re-write equation 2.4 such that for any vertex with tjin>0, we can express the effective latency τ¯ij as
τ¯ij=τij+δij'+tjin=τij+δij,
(3.2)
where δij' reflects the component of δij that does not account for the extra time due to tjin. δij=δij' modulo tjin. τij is a computed quantity dependent on sij and dij. But computing τ¯ij involves taking into account δij such that τ¯ij=τij+δij (see equation 2.4). δij affects when vj outputs or initiates a signal. The effect of tjin>0 is to delay by some amount when that occurs at time ti' (i.e., for a period of time tjin). Because δijR, it absorbs δij'.

Three scenarios fully describe the range of possible effects of an internal signaling delay. First, a signal Hj[vi]vj arrives at vj after which some amount of internal processing time given by tjinvj makes a decision to output a signal and does so instantaneously following that decision. Second, a signal Hj[vi]vj arrives at vj, which makes an instantaneous decision to output a signal, but it takes tjin before the signal from vj actually goes out. The third is a combination of scenarios 1 and 2 whereby some fraction tjin/A of tjin represents the time required to make a decision and tjin/B=tjin-(tjin/A) represents the time between when a decision is made and an output signal actually goes out. Note, of course, that independent of the magnitude of tjin, the ratio A/[B(A-1)]=1. From a practical perspective, however, we do not need to distinguish among these three conditions. From the perspective of the framework, if vj produces a signal due to Hj[vi]vj, then tjin will effectively cause an elongation of τ¯ij.

If for a specific system tjin<<τijandRj, it would have no effect on the dynamics. For any real physical system, tjin must always be finite and greater than zero, but we can safely ignore its effects if its timescale is much shorter than then the dynamics of the network. The computation of the refraction ratio as given does not change. But the effect on Λij could affect the global dynamics of the network if tjin is on the scale of τijandRj.

3.4  Geometric Dynamic Perceptrons: Summation from Fractional Contributions of Multiple Nodes

In this section, we describe the summation of fractional contributions of multiple winning nodes toward the activation of vj. Instead of a single node vi activating vj, whether deterministically or probabilistically, we now consider a running summation of contributions from a number of viHj(vi) adding up to a threshold value that then activates vj. Upon activation, vj becomes refractory, as previously described. Conceptually, this represents a competitive refractory model extension of the classical notion of a perceptron, whereby the metric considerations, and therefore signaling latencies, of the model play a critical role in determining node activation. These perceptrons can be thought of as having a geometric morphology or shape to account for computed latencies on the edges that represent the inputs into vj, similar to biological neurons (e.g., see Figure 1b). The interplay between the latencies and timing of discrete signaling events on the input edges, and the evolving refractory state of vj, determines the running summation toward threshold of signal contributions from arriving inputs. We assume weights and an activation function as per the standard perceptron model (see below). We also introduce a decay function that provides a memory or history for previous arriving signals, resulting in diminishing but nonzero contributions toward the summation from inputs that arrived at previous times relative to an observation time To. This is how we account for a relative contribution of signals given their temporal offsets. Thus, from the perspective of the framework and refraction ratio, the computational prediction being made is not which viHj(vi) will activate vj after To, but what subset of Hj(vi) will do so.

Consider a To, and assume that vj is not refractory. As a function of time, the running summation Σr from Hj(vi) must reach a threshold ΣT in order for vj to activate at some time tTo. Once activated, vj becomes refractory for a period Rj as usual. The specific contribution from one viHj(vi) will be the value of a weight (synaptic strength), wij. As is typical in a perceptron, we assume a set of weights associated with all incoming connections Wj={wij}. In our model, the maximum value of the contributing weight wij,max occurs at the time that the signal from vi arrives at vj after the end of the relative refractory period for vj, R¯j. This occurs at the time (τ¯ij-R¯j) (see theorem 2)—explicitly, wij(τ¯ij-R¯j) for a time-varying weight wij(t) that decays over time. Beginning at the next time step, we assume the contributing value of the weight starts to decay as a function of time: wij(t)<wij,max for times t>(τ¯ij-R¯j). In other words, there is a finite memory at future times to the arrival of a given signal, scaled to its weight value, that progressively decays over time to zero. This produces a complex and dynamic interplay between the latencies of discrete signaling events relative to each other (which in turn encodes an underlying geometry to the perceptrons and the network), the magnitude of the contribution from the respective weights once they do arrive, the kinetics of the decay of the contributing weights as a function of time, and the timing of the refractory state and recovery from the refractory state of vj. Excitatory versus inhibitory weights can be handled as discussed in section 3.2 in the sense that inhibitory weights have a negative impact on Σr.

Note also that this construction differs from the probabilistic extension of the deterministic version of the framework where one individual vi is capable of activating vj. For that, we assigned a probability distribution to the likelihood of activation of vj given a winning vertex Hj[vi]vj. Here, each viHj(vi) contributes a component to the running summation Σr. One could easily, however, extend the model further and construct a probabilistic fractional summation version by combining both approaches. This would add even further computational complexity to the total solution space represented by the combinatorial output of the network. However, we do not explicitly take this into account here.

We can again make use of the refraction ratios in the set Λijo to compute when Σr>ΣT with a decaying memory. Assume a nonrefractory vertex vj at some observation time To. We consider the value of the summation of weights into vj for all contributing viHj(vi). We want to compute what subset of Hj(vi) will result in a summation that exceeds the threshold value ΣT and at what time tTo this will occur. Once vj is activated, it becomes refractory for a period Rj. This will involve taking into account the weights Wj, values of τ¯ij for viHj(vi), and the value of R¯j at To.

For simplicity in the presentation, we first ignore the decaying memory function and ask how we can write an expression for Σr. Consider Λijo well ordered as in equation 2.6. Due to the ordering of the ratios, we can determine the ordering of the arrival of signaling events for each viHj(vi) at vj. This in turn allows us to determine the contributing weights wijWj in temporal order that sum to reach ΣT. This is the subset of Hj(vi) that fractionally contributes toward activating vj. We can index this subset ΛMΛijo={(Λij)mΛijo:m=1,2M} and then compute
Σr=m=1M(wij,max)mΣT
(3.3)
for the corresponding weights. This represents the subset of vertices that participate in activating vj, assuming that the summating contributing weights are persistently additive—in other words, they do not decay.
We can, however, take into account a temporal decay of each contributing weight by defining a function Di(t) that modulates each discrete signaling event following its arrival. When a signal from vi whose ΛijΛM first arrives at vj, Di(t)=0, such that there is no decay from wij,max. After some period of time ξ, we want wij=0, with progressively decreasing values in between. We assume Di(t) is a monotonically increasing function to unity so that the value of wij(t) progressively decays (eventually to zero) over some period of time. Formally,
Di(τ¯ij-R¯j)=0,
(3.4a)
Di[(τ¯ij-R¯j)+ξ]=1
(3.4b)
For(τ¯ij-R¯j)<t<(τ¯ij-R¯j)+ξiftn<tm
(3.4c)
thenD(tn)<D(tm),i.e.strictlyincreasing
(3.4d)
Then, for each individual contributing vi,
wij(t)=wij,max-wij,max·Di(t),
(3.5)
where, as above, the domain of Di(t) is (τ¯ij-R¯j)t(τ¯ij-R¯j)+ξ, and its codomain is 0Di(t)1. In the same way that we do not restrict the form of the activation function of vj, we intentionally define only a bounded but generalized Di(t) and do not explicitly restrict its form in any way. The actual rate of change and kinetics of the decay function will of course have a profound impact on the dynamics of contributing weights to Σr, but the exact form of Di(t) is allowed to vary.
Finally, we need to consider that Di(t) for individual contributing vi will start at different times after the observation time To. This reflects a time shift to when the decay for each wij begins relative to To. This shift determines the contribution of each wij toward Σr as time progresses. If we think of it in discrete time steps, for example, at each successive step, the values of wij from previously arrived signals will be progressively decaying as new signals arrive. Each wij will contribute different fractional amounts to Σr up to the point that some wMj,max due to a last-arriving signal from vmj results in ΣrΣT, producing the activation of vj. Notice how the arrival times of the subset of contributing vertices from Hj(i) that result in ΣrΣT, taken from (Λij)M all meet the condition that (τ¯ij-R¯j)<(τ¯Mj-R¯j). To compute the fractional contribution of any given wij at the moment threshold is reached, in other words, at t=(τ¯Mj-R¯j), we need to compute the value of its decay function at (τ¯Mj-R¯j) relative to when it started. The decay of the weight from its maximum value will have started after the signaling event arrives, so what we need to compute is how far along the decay is, the value of Di(t), at the time that reflects the difference from when the signal arrived and when threshold was reached, that is, at (τ¯Mj-R¯j). Thus, the individual fractional contribution of each wij,max can be computed using equation 3.3 by evaluating Di(t) at
t=(τ¯Mj-R¯j)-(τ¯ij-R¯j)=(τ¯Mj-τ¯ij).
Finally, the entire summation that includes all weight fractional contributions for the subset of vertices in Hj(vi) that achieve the activation of vj that belong to ΛM will be
Σr=m=1M(wij,max)m-(wij,max)m·Di(τ¯Mj-τ¯ij)ΣT.
(3.6)

Beyond the scope of this letter, we are exploring the construction and use of artificial neural networks constructed from geometric dynamic perceptrons. This work is part of the development of a fundamentally new machine learning architecture that does not necessitate statistical learning methods and requires no exposure to training data.

4  Visualizing the Dynamics with Feynman-Like Diagrams

In this section, we introduce two different versions of a diagram that provide a visual representation and summary of the dynamic model and signal summation process (see Figure 3). While it takes a bit of explaining to understand how each is constructed, they provide a compact way of visualizing the combination of processes involved and summarize a lot of information all at once. The well-known Feynman diagrams from particle physics are the inspiration for their structure, although this is immaterial for the discussion here. Note that we do not compute or show data for actual networks, since the focus of this letter is on the mathematics and theory of the model. However, the data computed by the algorithms are capable of generating these plots. Each version of the diagram summarizes the evolving temporal dynamics for a single Hj(vi) subgraph into a vertex vj (see Figure 2). There would exist one such diagram (of each version) for all the vertices in the network, that is, for all vjG(V,E). The entire global evolution of the network dynamics is then summarized by the set of all such diagrams. The intent here, however, is to introduce these diagrams as a visualization tool for the evolution of the local dynamics for a specific vertex vj. The difference between the two versions is that in Figure 3A, the entire causal history of Hj(vi) is preserved for all incoming vertices viHj(vi), while in Figure 3B the diagram shows the combined degree or amount of summation from all the vi vertices into vj at any given moment in time. We discuss each in detail.

Figure 3:

Feynman-like diagrams for visualizing the competitive refractory model and fractional summation of the signal summation process. (A) The entire causal history of Hj(vi) is shown for all incoming vertices viHj(vi). (B) A compactified representation of the dynamics showing the combined amount of summation Σr from all the vi vertices into vj at any given moment in time. See the text for details.

Figure 3:

Feynman-like diagrams for visualizing the competitive refractory model and fractional summation of the signal summation process. (A) The entire causal history of Hj(vi) is shown for all incoming vertices viHj(vi). (B) A compactified representation of the dynamics showing the combined amount of summation Σr from all the vi vertices into vj at any given moment in time. See the text for details.

4.1  Visualizing the Entire Causal History of the Dynamics

Figure 3A provides a visual record of the entire causal history of Hj(vi) for all incoming vertices viHj(vi) as it evolves in time—in other words, the ordered causal sequence of activation events. Time is represented on the y-axis, so the temporal progression of the dynamics evolves as one moves up the diagram. The initial moment of observation of the network is indicated as the horizontal dotted line at To. The collection of dots indicating vectors that begin at To represents a record of all signals already traveling on their respective edges of the Hj(vi) subgraph at the moment the network is observed. The x-axis represents the relative spatial positions of where traveling signals are relative to each other and the target vertex vj in a way we describe below.

We go through each part of the diagram in detail, with the paragraph numbers corresponding to the indicated numbering in Figure 3. For simplicity and clarity, we illustrate only excitatory inputs and do not show any inhibitory vectors in the diagram.

1. The vertical blue lines represent the state vj itself. In this example as drawn here, at time To, vj is partially refractory (with period of time R¯j remaining) from a signaling activation at some time prior to To. This is indicated by the length of the solid blue vector progressing forward in time. Once it stops being refractory, the solid blue vector changes to a dotted blue line, indicating vj is no longer refractory and capable of receiving and responding to subsequent incoming signals. Once signaling events again begin to arrive and summate, the dotted blue line changes to a squiggly blue line to show this. Eventually, once enough summation has occurred, an activation event occurs, and vj again becomes refractory for a period of time, switching back to a solid blue vector. Now we observe the full refractory period Rj. Of course, the initial state of vj at To for any particular diagram could start with any of these.

2. Each of the dots at t=To corresponds to a signaling event that is part way along its edge. The vectors that begin at these dots end when they reach vj. The slope of each vector provides a measure of the remaining time it takes the signal to arrive. One can read across to the time axis from the end of the vector once it hits the blue vj lines. The position along the spatial x-axis is relative in the sense that unlike the time axis, the origins of the vectors at each dot simply indicate how far away each signal is at To from the spatial position of vj relative to each other; more to the left of vj corresponds to signals that at To need to travel farther than signals farther to the right. Assuming a constant signaling speed or conduction velocity for all sij, as we have throughout the letter, the remaining time it takes signaling events that are already partially along their edges at the time the graph is observed at To will be proportional to the distance on the edge left to travel, |e¯ij|. The slope of each vector reflects the remaining edge distance and therefore time required for the signal to reach vj. Note that if the spatial axis was in real units, then the spacing between the origin of each vector would represent the actual physical distance of the signal from vj and assuming a constant sij for all signals every vector would be at 45 degrees. This is another valid way of drawing it. Here we chose to keep the spatial axis relative in arbitrary units to keep the diagram more compact and allow the slope to encode the spatial information that produces the remaining time the signals take to reach vj.

In the illustrative example here, we see that the first signal arrives while vj is still refractory and has no effect. The second signal begins a running fractional summation Σr. The next two signals add to Σr. The fourth arriving signal causes Σr>ΣT and vj fires, making it refractory again for the period Rj.

3. Any vector that begins at a time later than To represents a signal that left its origin from one of the vi vertices that connect into vj after the observation time To. Note how in the example for the vector shown here at position 3 in the diagram, the signal represented by this vector can represent either one of two possible conditions: (1) it originated at a node vi that did not have a signal partially traversing its edge at To, or (2) it corresponds to a signal from a vi that did have a partial signal at To and fired again. If condition 2, then the signal must have originated from a vertex vi that corresponded to one of the first two vectors (signals) to its right at To, that is, one of the two vertices closest to the spatial position of vj at To, since its total edge length |eij| cannot be shorter than a partial edge distance of |e¯ij| at To. To be fair, it is not possible to distinguish these two conditions from the diagram as drawn without a vertex label. However, the algorithms readily keep track of this too if necessary. This diagram is able to visualize the dynamics in a way that summarizes a lot of information that can be quickly gleamed.

4. Finally, in position 4 in the diagram, we show two signals originating from the same vi but occurring at different times. Because we assume that sij is constant, the slopes for both vectors, which encode the time it takes for these signals to travel the full |eij| for a particular vi, will be the same.

4.2  Visualizing the Dynamics of the Summations Σr

In Figure 3B, there is less information in the sense that the signaling events that lead to the activation of vj are compactified into a single vector. But the result is the ability to intuitively visualize the progression of the running fractional summation Σr that leads to the activation of vj. The time axis on the y-axis remains the same, but the x-axis is no longer the relative spatial position of the signals from vertices viHj(vi) that make up the subgraph. Rather, it is the degree or amount of Σr itself. The fractional summation culminates at the vertical dotted line indicating the threshold summation value required to trigger an activation or firing of vj at ΣT:

1. Vectors in black show an accumulation or increase in Σr. Here, they represent the totality of summations producing an increase in Σr. This does not mean only the summation of excitatory inputs adding to Σr but also counteracting inhibitory inputs that decrease or delay Σr (depending on how inhibitory inputs are accounted for). Importantly, a component of this vector is the decay of weight contributions for both excitatory and inhibitory signals as determined by the decay equation 3.4. Black vectors that increase Σr imply that excitatory summations outpace inhibitory summations that detract from Σr due to a combination of the number of excitatory inputs or frequency of excitatory signals.

2. Red vectors are the same as black vectors but represent the opposite process: net inhibitory summations that are greater than excitatory summations. This then reduces the value of Σr.

3. Through these processes, there will be a back-and-forth fluctuation of summating events that push and pull Σr to and away from ΣT. This is the model equivalent of and directly analogous to subthreshold membrane potential fluctuations in the dendritic trees of biological neurons. In real neurons, spatial and temporal summation in dendrites attempt to reach the membrane threshold potential at the initial segment in order to trigger an action potential. As discussed at the beginning of the letter, this is in effect what our framework models and abstracts out from the neurobiology. It represents the main motivation for this work. In the example shown here at position 3 in the diagram, an inhibitory summation process that is lowering Σr all of a sudden reverses because the summation of excitatory events overtakes it and starts driving Σr toward ΣT.

4. Here an activation event is illustrated. Σr reaches ΣT. A red vector must follow for at least a period corresponding to Rj due exclusively to the decay function driving down Σr, since vj will not be able to respond to inputs, excitatory or inhibitory.

5. Once Rj ends, if the summation conditions are right, Σr will begin to increase again. But note how the red vector can continue longer than the period dictated by Rj if summation conditions are not favorable.

It should be clear how the multiple summating vectors in panel A that trigger an activation at vj correspond to the equivalent compactified perspective in panel B. The entire mathematical description introduced in the preceding sections can be visualized in these two ways. In subsequent work, we will explore the use of these Feynman-like diagrams to study how the local rules that drive the dynamics of the individual Hj(vi) subgraphs that these diagrams represent produce the global behaviors and dynamics of the overall network.

5  Efficient Signaling

In this section, we prove a notion of optimized efficient signaling in the context of the framework and the refraction ratio. These arguments emerge naturally from a consideration of how the ratio is defined. For convenience, we limit our discussion to the deterministic version of the framework. Given an effective refractory period R¯j and effective latency τ¯ij along an edge eij, Hj[vi]vj, is dependent on the limτ¯ijR¯j+ for Λij=R¯j/τij¯—in other words, the first discrete signal that arrives at vj after it stops being refractory. Intuitively, efficient signaling in the context of the framework means that there should be a temporal match between the arrival time of the signals relative to how quickly nodes can internally process signals. When a mismatch between these two considerations occurs, it can cause a breakdown in the signaling dynamics of the network. For example, we have previously shown numerically (in computational simulations) that if signaling speeds sij are too fast or, equivalently, if the latencies τ¯ij are too short, compared to the amount of time a node requires to process a signal and recover to a state in which it can respond again to another input, the network will not be able to sustain internal recurrent activity (Buibas & Silva, 2011). This reflects a form of inefficient signaling in the sense that the network requires constant external driving (energy) to sustain it.

Conversely, if sij is too slow or the set of τ¯ij too long, the network will be inefficient in a very different sense. It has the potential for faster dynamic signaling that is not being realized. Time, as a resource, is being wasted in such a network. The following discussion formalizes these concepts.

We first derive upper and lower bounds for the refraction ratio between connected vertices vi and vj. We argue that these bounds formalize a notion of optimal efficient signaling that reflect a match between the rate of discrete signal propagation (information flow) between vertices relative to the internal dynamics of the vertices.

Theorem 3
(Optimized Refraction Ratio Theorem). Let G=(V,E) represent a geometric network consisting of subgraphs Hj(vi). For each vivj vertex pair with a signaling speed sij between vi and vj, the optimal refraction ratio [Λij]opt at an observation time To is bounded by
[Λij]opt=limτijRj+Λijwhenφjandδij=0[Upperbound],
(5.1a)
[Λij]optlimδij-φjΛijwhenφj=Rj[Lowerbound].
(5.1b)
Given these bounds then, an optimized refraction ratio will be such that
[Λij]opt=limτijR¯j+R¯jτij1.
(5.1c)
Proof.

The necessary condition for the activation of vj by viHj(vi) is τ¯ij>R¯j. By equation 2.3R¯j=Rj-φjwhere0φjRj, which implies that 0R¯jRj. R¯j is bounded by its very construction. The absolute lower bound on R¯j implies that activation of vj by a viHj(vi), τ¯ijR¯j+, will be achieved when τ¯ij0+ (i.e., is just slightly larger than zero) and the absolute upper bound implies that τ¯ijRj+. But note how τ¯ij can always achieve these bounds independent of τij for a given vivj pair at To because by equation 2.4, δijR; any vertex vi can activate vj independent of the absolute latency τij by delaying the initiation of an output signal at vi long enough if τij is too short or initiating a signal at vi prior to To if τij is too long. However, by equation 2.6 and theorem 1, τ¯ij need only be slightly larger than R¯j in order to successfully signal vj (i.e., τ¯ijR¯j+). Because R¯j is naturally bounded by 0R¯jRj, it follows that the optimal signaling condition will be given by τijR¯j+ for values of δij not much smaller than zero or not much greater than zero in order to meet the condition that τijR¯j+ while avoiding compensation by δij. In other words, the response dynamic range for any vj will always be bounded by the limits of R¯j in the sense that these limits determine the temporal properties of when vj can actively participate in network signaling and when it cannot. Ultimately, of course, this is a function of vj's internal dynamics, which in turn determines Rj and R¯j. No value of τij need be much greater than Rj for any vi with a directed edge eij into vj. When the condition τijR¯j+ is met, it ensures that such a τij is guaranteed to be able to operate over the entire response dynamic range of vj (i.e., all values of R¯j). Given this condition for optimized signaling in the context of the refraction ratio, then, for the upper bound, this optimized boundary condition will occur when τijRj+ (when φj and δij=0) because it represents the upper achievable limit for R¯j (when φj=0) and forces the optimal condition that τijRj+ without compensating with δij.

For the lower bound, the optimal condition is given by τ¯ij0+δij-φj when φj=Rj, since when φj=RjR¯ij=0. Forcing the condition that φj=Rj implies that τij on its own is capable of meeting the lower bound without compensation by δij. We can define an optimized bound as |τij-Rj|<ε for some bounded error ε. If τ¯ij is too short, either because the path length of eij is too short or sij is too fast, this implies that given Rj, τ¯ijRj+ if δij0. To achieve the lower bound, it would require δij<0 so that τ¯ij<τij. To achieve the upper bound, it would require δij>0 so that τ¯ij>τij. If τ¯ij is too long, either because the path length of eij is too long or sij is too slow, this implies that given Rj, τ¯ij0+ if δij-φj. When these constraints are met, it ensures that limτijR¯j+R¯j/τij1. For the lower bound, the important condition is that τ¯ij0+ when δij-φj. It is trivial what δij is, since this condition will always be met when δij=-τij. But for the upper bound the important condition is that τijRj when δij0, which implies that in every case τ¯ijτij.

What this implies is that if vi satisfies the bounding conditions, it will always be within a range where it could efficiently “win” and activate vj for any value of R¯j at an observation time To and time of signaling initiation by vi at ti. By efficient, we mean without forcing δij to compensate. The given vi may not of course always “win” in activating vj, but it is assured to be as efficient as possible—as efficient as any other node in its signaling of vj over all values of R¯j. If Rj is the same for all nodes in the network, Rj=RjG(V,E), this in effect bounds the dynamic window over which all network dynamics—that is, all temporal information (signaling) processes—should occur on that network. Explicitly, this dynamic window is given by R¯j=Rj-φjfor0φjRj, so there is no need or reason for any τij to go far beyond this dynamic window in order to satisfy the optimality condition τ¯ijR¯j+. This is in essence the intuitive idea of using the bounds derived in theorem 3 to define optimized signaling or information flow between node pairs. Note that we must keep the explicit condition that φj=Rj because that forces Λij=0 only when δij-φ+. Otherwise, in the general case, any value of τ¯ij>>0 will result in Λij=0 for any value of R¯j.

We can also prove the following corollaries.

Corollary 1.
Given the conditions for lower and upper bounds in theorem 3, let [δij]upper denote the value that δij must take in order to achieve the optimal upper-bound condition for some value of τij. Similarly, let [δij]lower denote the value that δij must take in order to achieve the optimal lower-bound condition. In every case, the relationship between [δij]upper and [δij]lower is given by
[δij]lower=[δij]upper-Rj.
Proof.

The condition for the upper bound is τijRj+ for δij=0 (and φj=0). Since τ¯ij=τij+δij, if τijRj, then τ¯ijRj+. For a given τij and for a known or measurable Rj, [δij]upper=-(τij-Rj), since this is what the value of δij would have to be in order to achieve the optimal condition. For the lower bound, τ¯ij0+ when δij-φj given that φj=Rj. Thus, if τij-φj=τij-Rj>0 or, more correctly, if τ¯ij0+, it implies that [δij]lower=-τij would be needed to meet the lower-bound optimality condition. The difference between [δij]lower and [δij]upper is therefore -τij-[-(τij-Rj)]=-Rj. Thus, [δij]lower=[δij]upper-Rj.

Corollary 2.

If a signal viHj(vi) characterized by a latency τij on the edge eij is able to achieve either the optimal upper bound or optimal lower bound as per theorem 3, then it is guaranteed to be able to achieve the other optimal bound.

Proof.
Asking if a signal capable of achieving the upper bound can also achieve the lower bound is equivalent to asking if τ¯ij0+ when τij=Rj. But the condition for the lower bound is τ¯ij0+ when δij=-φj. Substituting for these explicit variables, we arrive at
τ¯ij=τij+δij,=Rj-φj,
but since φj=Rj for the lower bound, it implies that τ¯ij=0 or, more appropriately, τ¯ij0+.
Asking if a signal that satisfies the optimality condition for the lower bound can also achieve the upper bound is equivalent to asking if τ¯ijRj+ when δij-φj. Similarly,
τ¯ij=τij+δij,0=τij-Rj,τij=Rj,
which implies that the optimality condition for the upper bound is satisfied.

A strict definition of an optimally efficient network follows. In every case, as a function of the effective refractory period R¯j and effective delay time τ¯ij along the edge eij, the condition for the winning vertex vi that achieves activation of vj (i.e., Hj[vi]vj) is dependent on the limτ¯ijR¯j+viG(V,E). When this condition is satisfied for all edges eijE={eij}, for all vivj node pairs by the upper- and lower-bound definitions for [Λij]opt in theorem 3 (see equation 3.16) such that Λij=R¯j/τ¯ij1, the network is optimally efficient (i.e., [Λij]optviG(V,E)). This is equivalent to requiring the condition |τij-Rj|<εviG(V,E) for some arbitrarily small value of ε.

A consequence of this is that for a network to meet this strict definition, it must exhibit a lattice structure. Given constant values of |tj| and Rj and sijeijE, optimized efficient signaling at the network scale is achievable only when dij=|eij|=CeijE, where C is a constant. Geometrically, this implies that the network must exhibit a lattice structure. To see this, note that if we assume optimal signaling between any arbitrarily chosen but specific pair of vertices vI,vJ such that ΛIJ1+, then
Λij=RJ-φJτIJ-δIJ=RJ-φJdIJsIJ+δIJ1+.
For the lower-bound φi and δij=0, so the condition for optimal signaling will be Rj·(sij/dij)1+. This means that for any vivj pair such that dijdIJ, Λij1+.
For the upper-bound φj=δij, so
Λij=Rj-δijdijsij+δij1+.
We know that [δij]lower=[δij]upper-Rj:=K. Then
Λij=Rj-Kdijsij+K1+.
Then evaluating again for the lower bound given the substitution yields the same result as above, such that Λij1+. Of course, while this provides a strict criterion of optimized network signaling by evaluating the optimality of signaling across all node pairs independently, it is not necessarily equivalent to optimized network dynamics. In other words, a pure lattice structure may not be (and in most cases will not be) the best geometric structure to optimize the dynamical flow of information across a network in support of some function or behavior or learning the network is intended to do. This condition is strictly mathematical as a consequence of theorem 3.

6  Conclusion

In this letter, we presented an intuitively simple framework that describes the competing dynamics of signaling and information flows in spatial-temporal (geometric) refactory networks derived from foundational principles of biological cellular neural signaling. These results provide a systematic explanation for how the interplay between strictly local geometric and temporal process at the scale of individual interacting nodes ultimately gives rise to the global behavior of the network. At the core of the model is the assumption that the response of each vertex is causally independent from whatever all the other vertices in the network are doing, even though collectively, they contribute to the global dynamics. The internal (to each vertex) determination of the state yj is not dependent on any average metric of the state of the network as a whole or on statistical probability densities associated with the frequency of occurrence of events such as in Markovian processes. This allows us to model and simulate network dynamics by keeping track of the latencies associated with signals on the edges without needing to account for or model any number of dynamic variables necessary to describe the internal models of the individual nodes that we may have little to no information about—or necessitating the computation of statistical variables from which network dynamics is then inferred, which generally precludes real-time analysis due to the need of observing or measuring sufficient data first. This is in direct contrast to most other network analysis approaches that attempt to measure or capture global properties.

The class of networks we define and investigate here can be thought of as belonging to a broader class of networks refered to as spatial-temporal and diffusion networks. These networks are of relevance to many natural, engineering, and technological systems, because the conceptual model they provide naturally maps to many real-world physical systems (Batagelj, Doreian, Ferligoj, & Kejzar, 2014; Cencetti, Clusella, & Fanelli, 2018; George & Kim, 2013; Gomez-Rodriguez, Balduzzi, & Schölkopf, 2011; Gong et al., 2018; Shanmugam, Mani, Rajan, & Joo, 2018). Examples include transportation systems, communication networks, social networks, the spread of infectious diseases (including computer viruses and malware), physical-chemical systems such as the interactions of particles in solution and diffusion, and -omics-type biological networks of molecular and genetic interactions. However, much of the literature on these classes of networks has focused on descriptive theory, definitions of concepts and notation, and various functional metrics. There are comparatively fewer applications, algorithms, and testable predictions. There is an almost exploratory quality to the existing literature. In particular, there is no work we are aware of that develops a theoretical foundation that takes advantage of canonical neurobiological processes to arrive at a set of practical algorithms, as we do here, with applicability to both neuroscience and machine learning. We have previously argued that theoretical neuroscience, while challenging, offers an opportunity for a deeper understanding than purely numerical or simulation-based computational neuroscience models (Silva, 2011).

As an example of a real-world application of the ideas and framework we discuss here, of particular interest to us is the signaling dynamics responsible for computation in biological neural networks across scales of organization (e.g., dendritic trees, neural circuits, or the interaction between different brain regions) constrained by the underlying structure—in other words, the structure-function relationships responsible for neural dynamics. We recently showed that basket cell neurons in the cortex are putatively designed to optimize for the refraction ratio. They seem to be designed to optimize the ratio between the refractory period of the membrane and action potential latencies along the axon between the initial segment (where action potentials begin) and the synaptic terminals (where they end; Puppo et al., 2018). Dynamic signaling on branching axons is critical for rapid and efficient communication between neurons in the brain. Efficient signaling in axon arbors depends on a trade-off between the time it takes action potentials to reach synaptic terminals (temporal cost) and the amount of cellular material associated with the wiring path length of the neurons' morphology (material cost). However, where the balance between structural and dynamical considerations for achieving signaling efficiency is, and the design principle that neurons optimize to preserve, this balance has remained elusive. We took advantage of the theoretical prediction of theorem 3 and went looking to see if biological neurons displayed optimized signaling in the context predicted by the refraction ratio. One interpretation of our results is that the convoluted paths taken by axons reflect a design compensation by neurons to slow down, signaling latencies in order to optimize the refraction ratio. This was not a serendipitous discovery. We went looking to see if neurons actually optimize the refraction ratio because the mathematics pointed us in that direction.

Acknowledgments

I am grateful to Fang Chung (Department of Mathematics, UC San Diego) for her input and feedback when this work was in its earliest stages and to Vivek George (Department of Bioengineering, UC San Diego) for many fruitful discussions. I sincerely acknowledge the effort of the two anonymous reviewers. This was not an easy manuscript to review. Their very thoughtful feedback has strengthened the final work product. This work was supported by grants 63795EGII and N00014-15-1-2779 from the Army Research Office (ARO), U.S. Department of Defense, and in part from unrestricted funds to the Center for Engineered Natural Intelligence.

References

Batagelj
,
V.
,
Doreian
,
P.
,
Ferligoj
,
A.
, &
Kejzar
,
N.
(
2014
).
Temporal and spatial networks
. In
V.
Batagelj
,
P.
Doreian
,
A.
Ferligoj
, &
N.
Kejžar
, (Eds.),
Understanding large temporal networks and spatial networks
.
Chichester, UK
:
Wiley
.
Buibas
,
M.
, &
Silva
,
G. A.
(
2011
).
A framework for simulating and estimating the state and functional topology of complex dynamic geometric networks
.
Neural Computation
,
23
,
183
214
.
Cencetti
,
G.
,
Clusella
,
P.
, &
Fanelli
,
D.
(
2018
).
Pattern invariance for reaction-diffusion systems on complex networks
.
Nature Scientific Reports
,
8
,
16266
.
George
,
B.
, &
Kim
,
S.
(
2013
).
Spatio-temporal networks: Modeling and algorithms
.
Berlin
:
Springer
.
Gomez-Rodriguez
,
M.
,
Balduzzi
,
D.
, &
Schölkopf
,
B.
(
2011
).
Uncovering the temporal dynamics of diffusion networks
.
arXiv, 1105.0697
.
Gong
,
J.
,
Liu
,
X.
,
Liu
,
T.
,
Zhou
,
J.
,
Sun
,
G.
, &
Tian
,
J.
(
2018
).
Dual temporal and spatial sparse representation for inferring group-wise brain networks from resting-state fMRI dataset
.
IEEE Transactions on Biomedical Engineering
,
65
,
1035
1048
.
Kandel
,
E.
,
Schwartz
,
J. A.
,
Jessell
,
T. M.
,
Siegelbaum
,
S. A.
, &
Hudspeth
,
A. J.
(
2012
).
Principles of neural science
(5th ed.),
New York
:
McGraw-Hill
.
Magee
,
J. C.
(
2000
).
Dendritic integration of excitatory synaptic input
.
Nature Reviews Neuroscience
,
1
,
181
190
.
Manor
,
Y.
,
Koch
,
C.
, &
Segev
,
I.
(
1991
).
Effect of geometrical irregularities on propagation delay in axonal trees
.
Biophysical Journal
,
60
,
1424
1437
.
Minsky
,
M. L.
(
1967
).
Computation: Finite and infinite machines
.
Upper Saddle River, NJ
:
Prentice Hall
.
Puppo
,
F.
,
George
,
V.
, &
Silva
,
G. A.
(
2018
).
An optimized structure-function design principle underlies efficient signaling dynamics in neurons
.
Nature Scientific Reports
,
8
,
10460
. doi:10.1038/s41598-018-28527-2.
Rosenblatt
,
F.
(
1958
).
The perceptron: A probabilistic model for information storage and organization in the brain
.
Psychological Review
,
6
,
386
408
.
Shanmugam
,
L.
,
Mani
,
P.
,
Rajan
,
R.
, &
Joo
,
Y. H.
(
2018
).
Adaptive synchronization of reaction-diffusion neural networks and its applications to secure communication
.
IEEE Transactions on Cybernetics
. doi:10.1109/TCYB.2018.2877410.
Silva
,
G. A.
(
2011
).
The need for the emergence of mathematical neuroscience: Beyond computation and simulation
.
Frontiers in Computational Neuroscience
,
5
,
51
. doi:10.3389/fncom.2011.0005.