We describe the construction and theoretical analysis of a framework derived from canonical neurophysiological principles that model the competing dynamics of incident signals into nodes along directed edges in a network. The framework describes the dynamics between the offset in the latencies of propagating signals, which reflect the geometry of the edges and conduction velocities, and the internal refractory dynamics and processing times of the downstream node receiving the signals. This framework naturally extends to the construction of a perceptron model that takes into account such dynamic geometric considerations. We first describe the model in detail, culminating with the model of a geometric dynamic perceptron. We then derive upper and lower bounds for a notion of optimal efficient signaling between vertex pairs based on the structure of the framework. Efficient signaling in the context of the framework we develop here means that there needs to be a temporal match between the arrival time of the signals relative to how quickly nodes can internally process signals. These bounds reflect numerical constraints on the compensation of the timing of signaling events of upstream nodes attempting to activate downstream nodes they connect into that preserve this notion of efficiency. When a mismatch between signal arrival times and the internal states of activated nodes occurs, it can cause a breakdown in the signaling dynamics of the network. In contrast to essentially all of the current state of the art in machine learning, this work provides a theoretical foundation for machine learning and intelligence architectures based on the timing of node activations and their abilities to respond rather than necessary changes in synaptic weights. At the same time, the theoretical ideas we developed are guiding the discovery of experimentally testable new structure-function principles in the biological brain.
We describe the construction and theoretical analysis of a framework that models how local interactions among connected node pairs can be used to compute the global dynamics of a class of spatial-temporal network. By analyzing the local dynamics of incident signals into nodes and how signals compete to activate downstream nodes, we were able to prove a number of properties associated with the dynamics of the network. This framework was derived from a theoretical abstraction of the canonical neurophysiological principles of spatial and temporal summation in biological neurons (Magee, 2000; Kandel, Schwartz, Jessell, Siegelbaum, & Hudspeth, 2012). We considered spatial-temporal networks in the sense that we analyzed signals along (geometric) directed edges in a metric space that give rise to temporal latencies or delays of signals. The metric space is not intended to conform to any specific coordinate system. The only requirement is that it be able to support a spatial embedding that preserves geometric distances between edges, because it is the geometry of the edges that gives rise to the signaling latencies given a conduction velocity (signaling speed). When combined with a refractory state for each node, a central concept in this model, the interaction between the two has a dominating effect on the global network dynamics. Following successful activation, every node experiences a subsequent period, or state, of refractoriness, during which it is incapable of responding to subsequent inputs. The framework we developed reflects local processes from which global behaviors emerge.
At its core, our model takes into account how the timing of different signals influences their competition to activate target nodes they connect into, given that the internal (refractory) state of the node they are competing for may or may not allow that node to be activated. The refractory period is a value that reflects the internal state of the node in response to its activation by an upstream winning node (or set of nodes). In contrast to most other work, we do not assume anything about the internal model that produces this refractory state. For example, this could include an internal processing time during which the node is making a decision about how to react or an internal reset period after the node has reacted before it is in a state capable of responding to new inputs. A refractory period is a reasonable assumption for any physically constructible network, since infinitely fast processing times are not realizable. If the timescale of such a refractory state is much shorter than the time it takes for propagating signals to reach their vertices, due to either fast signaling speeds or short edges, it can have a significant affect on network dynamics and function. The inverse will also affect the dynamics, where the refractory period is much longer than the time it takes for signals to arrive at target nodes. These concepts are foundational to the analysis we present in this work.
In order to intuitively capture the relationship between signal flows on edges and the refractory states of the participating nodes, we defined a refraction ratio for each node. For every pair of connected nodes in the network, we define this as the ratio between the amount of time left in the recovery from the refractory period for the node receiving a signal, relative to the amount of time before the next signaling event arrives at that node from the upstream node that connects into it. The refraction ratio captures the relationship between the speed and temporal delays of signaling or information flow, which is bounded by the spatial geometry of the edges, and the refractory state of the node. We use the term activation here to imply an appropriate reaction of that node to the signal it has received. In the case of biological neurons, for example, it would be the generation of an action potential that propagates down the axon and axonal arborizations to synaptic terminals. In neurons, the membrane refractory period due to the biophysics of the membrane ensures the directional nature of the traveling action potential wave along the axon and sets the cell's maximal firing frequency. Analogously in our framework, it is the generation of discrete signaling events that directionally travel between edges that connect the nodes in a network. As we and others have previously shown, the interplay between temporal latencies of propagating discrete signaling events, relative to the internal dynamics of the individual nodes, can have profound effects on the dynamics of a network (Buibas & Silva, 2011; Manor, Koch, & Segev, 1991; George & Kim, 2013). In a spatial-temporal network, as we define it in this work, temporal latencies need not be directly assigned but result from the relationship between signaling speeds (conduction velocities) and the geometry of the edges on the network (i.e., edge path lengths).
An analysis of the framework led to two main results. First, a series of systematic extensions of the basic construction resulted in a spatial-temporal version of a perceptron. In the classical perceptron, contributing weight summations from upstream nodes into an activation function determine whether that node produces an output signal (Rosenblatt, 1958; Minsky, 1967). In our case, we considered how temporal latencies produce offsets in the timing of the summations of incoming discrete signaling events. Signaling latencies affect when and how the threshold of the perceptron's activation function is reached as a function of a running summation in time. This produces a much richer dynamical repertoire relative to the classical perceptron model. As part of this model, we defined a decaying memory function such that the maximum value of the weight for a given connection occurs at the time of arrival of the signal. Subsequent time steps result in progressive decreasing (i.e., decaying) contributions from the maximum weight value to the summation function.
Second, a theoretical result from the analysis of the refraction ratio was a set of bounds that allowed us to formally define a notion of efficient signaling within the context of the framework. We show that an optimal ratio is one where the timing of information propagation between connected nodes does not exceed the internal dynamic timescale of the nodes. In other words, it represents a balance between how fast signals propagate through the network relative to the time needed for each node to process incoming signals. Efficient signaling in the context of the framework we develop here means that there needs to be a temporal match between the arrival time of the signals relative to how quickly nodes can internally process signals. A mismatch between these two considerations can cause a breakdown in the signaling dynamics of the network. We have previously shown in numerical experiments that a qualitative imbalance of this phenomenon can result in the breakdown of a network to sustain recurrent activity (Buibas & Silva, 2011). Intuitively, efficient signaling as we develop it in this work implies that there is necessarily a temporal match between the amount of time it takes signals to travel between connected node pairs in a network relative to how quickly the nodes receiving signals can internally process them. A mismatch in time between these two considerations results in a breakdown in the signaling dynamics of the network. It is this notion of efficient signaling that we formalize here.
The focus of our exposition in this letter is the development and theoretical analysis of the framework, not a numerical or computational investigation. We also intentionally do not show any applications of the work, which are beyond the intent of this letter. Ongoing work is using the framework to develop a new machine learning architecture. This architecture is fundamentally different from existing artificial neural network models in how it learns and encodes information and data. We are also using the refraction ratio in the neurophysiological analyses of structure-function dynamics in biological neurons and networks. Using high-resolution morphological reconstructions of axon shapes, we recently showed that the refraction ratio reflects a design principle that biological neurons optimize. It reflects a balance between the wiring lengths of axons (material cellular costs) versus signaling speeds (temporal costs; Puppo, George, & Silva, 2018).
The letter is organized as follows. In section 2 we introduce the basic construction of the competitive refractory framework and provide an analysis of its dynamics. Following a set of preliminary definitions in sections 2.1 and 2.2, we introduce the refraction ratio in section 2.3. We then show in section 2.4 how we can use this ratio to compute the set and order of winning nodes in parallel across a network, essentially providing an algorithm to compute global behaviors from local dynamics at the scale of individual node pair interactions. Section 3 outlines a number of systematic extensions of the basic construction. In section 3.1, we discuss a simple probabilistic extension of the deterministic version of the framework. Section 3.2 introduces inhibitory inputs into a node (in contrast to excitatory inputs). In section 3.3, we discuss the role and contribution of internal node processing times to subsequent signaling latencies on edges. Section 3.4 concludes with the development of fractional node contributions summating in time in order to reach an activation threshold that activates a downstream node. This is our model of a geometric dynamic perceptron. Section 4 introduces two distinct versions of graphics derived from Feynman diagrams intended to provide a visualization tool for the dynamics. In section 5, we introduce a notion of optimal efficient signaling within the context of the framework. We define a set of bounds the refraction ratio must conform to. Section 6 provides some concluding comments.
2 Competitive Refractory Framework
We considered the geometrical construction of a spatial-temporal network in the following sense. We assume that signals or discrete information events propagate between nodes along directed edges at a finite speed or conduction velocity, resulting in a temporal delay or latency of the signal arriving at the downstream node. Imposing the existence of signaling latencies implies a network that can be mapped to a geometric construction, where individual nodes could be assigned a spatial position in space in for an ordered triplet for each vertex for all vertices , where is the number of nodes and therefore the size of the network. We use the terms vertices and nodes interchangeably throughout the letter. Formally, vertices belong to a graph that models a particular network, with the nodes belonging to the network. Directed edges connecting node pairs could have a convoluted path, that is, a Jordan arc. There is no restriction that edges have to be spatially minimizing straight-line edges (see Figure 1). A signaling latency expresses the ratio between the distance traveled on the edge, , relative to the speed of the propagating signal , between a vertex that connects into a vertex . For all pairs of connected vertices , . While one does not have to explicitly consider and , the existence of signaling latencies can always be mapped to these variables. This is analogous to the conduction velocity of action potentials traveling down the convoluted axon and axonal arborizations of a biological neuron (see Figure 1B).
The set of all edges in the graph is given by , where denotes the directed edge from vertex to vertex . We define the subgraph as the (inverted) tree graph that consists of all vertices with directed edges into . We write to represent the set of all vertices in and to refer to a specific . We assume there exist discrete signaling events traveling at a finite speed on the edge . The signaling speed from to is bounded such that ; that is, it must be finite. We introduce the notation to mean a vertex that causally leads to the activation of . In other words, this notation indicates the “winning” vertex whose signal manages to activate (see Figure 2).
We then define an absolute refractory period for the vertex by . This reflects the internal dynamics of once a signaling event activates it—for example, the amount of time the internal dynamics of requires to make a decision about an output in response to being activated or some reset period during which it cannot respond to subsequent arriving input signals. We place no restrictions on the internal processes that contribute toward . We assume that we do not know and cannot observe the internal model of , which could be quite complex (e.g., a family of temporal differential equations that produce its refractory period). But we do assume we can observe or measure how long is. We also assume that , that is, there cannot exist an infinitely fast or instantaneous recovery time, even though it can be arbitrarily short. This is a reasonable assumption for any physically constructible network.
Consider a vertex with a directed edge to a vertex . For to signal , there must be some discrete physical signal representing a flow of information from to over the edge that connects them. This signal must travel at some finite speed . could be a constant value for all edges, but this need not be true in the general case. Similarly, if all nodes in a network share the same internal dynamics, then . But the framework does not assume this and can accommodate differing node-specific values of the refractory period. Once receives a signal from it becomes refractory for a period and will not be able to respond to another incoming signal during this period of time. The temporal nature of implies that as time progresses, it shortens and eventually decays to zero, at which time is able to respond to another input.
2.2 Internal Dynamics of
Once the winning vertex “activates” , it becomes refractory for a period of time during which , determined by its internal dynamic model. An important point is that if the state of at some arbitrary observation time is , it could remain refractory for some time if it had become refractory prior to . This situation is interesting because we have to take into account phase shifts in and at in order to understand the patterns of node activations—in other words, which arriving signal results in the activation of is dependent on the amount of the refractory period of remaining relative to the time of arrival of the signal. We call the amount of residual refractory period at the effective refractory period, which we write as . This is at the core of our model.
Algorithmically, we can compute at discrete times in parallel for every vertex in the network (i.e., every ), which causally activates (i.e., ). We will achieve this by keeping track of the temporal interplay of propagating signaling events on the edges relative to the refractory states of the individual vertices by computing the refraction ratio for all node pairs.
2.3 The Refraction Ratio
Before proceeding further, we point out a number of trivial and unallowable conditions that are necessitated by the physical construction of real-world networks and the definitions above. The trivial lower bound occurs as , at all times. But recall that . implies a nonrefractory vertex capable of instantaneous recovery to an incoming signal from an upstream vertex, a condition that is not physically realizable. The trivial upper bound occurs as , at all times, in which case there would be no information flow or signaling ever. As , becomes undefined, which is equivalent to stating since for a fixed signaling speed . Equivalently, if . But these conditions are unattainable. is necessarily restricted to finite dynamic signaling and information flow in a network.
Intuitively, for any into when , if every initiates a signal at exactly the same moment and is not refractory, the vertex with the shortest edge path will win and activate . In other words, assuming a constant signaling speed for all if we let be the set of all edge paths for , then will be achieved by for .
Under realistic conditions, at an arbitrary observation time , there is likely to be a temporal offset between when each signals and how far along is in its recovery from its refractory period due to a previous signaling event. Furthermore, each is statistically independent from every other , so that the amount of temporal offset for each vertex signaling will be different. In order to compute these offsets and keep track of the overall dynamics of the network, we need to index two different notions of time. We define to be the moment at which initiates a signal along its edge toward . The other is the observation time itself, which is the moment at which we observe or measure the state of . Put another way, it is the moment in time at which we chose to interrogate how far along each signaling event is on its respective edge . Such an offset in the progression of discrete signaling events along different edges is the general case, except in the special case when every signals at the same moment.
The extra degrees of freedom that result from the temporal offsets in the timing of arriving signaling events at for the set relative to each other produce a much larger combinatorial solution space for how may be activated, although we do not systematically fully investigate the size of this space in this letter. The global dynamics of the network results from the local statistically independent dynamics of each vertex and its corresponding . Once is activated, it contributes to the dynamics of the activation of the downstream vertices it connects into. The activation of results in its contribution as one of the for each of the it is a part of.
2.4 Analysis of the Refraction Ratio
Intuitively, the “winning” vertex that successfully achieves the activation of , , will be the first signaling event that arrives at immediately after has stopped being refractory. This is equivalent to stating that will be achieved by the with the smallest value of larger than . This condition guarantees . This will be the case for the value of such that (i.e., approaches from the right)—that is, is slightly longer than . The analysis of will therefore necessitate computing the order of arriving signaling events to determine which one meets this condition first. This is effectively an algorithm that computes at .
Let the set be defined as in equation 2.6. When , it guarantees that . This means that the effective refractory period of is either partially recovered from the absolute refractory period at the observation time if , or else when (see equation 2.3). If is still refractory for some period , the signal with the smallest may not be the signal that activates because it may arrive before ends. In this case, the signal that arrives first immediately after ends will satisfy . Because of how the refraction ratio and the set are defined, with (see equation 2.5), will be met by the vertex that satisfies , where the notation indicates the ceiling function, as is typical. To see this, assume some constant value . Let for two and with . Then . The closest of the two values of to '' then is . For a set , the condition first relative to all is the largest ratio closest to unity, that is, .
If , then , and the “winning” vertex will be given by , since only when and . In other words, if is not refractory at then the first signaling event that reaches it is guaranteed to activate . This will be given by the trivial condition of the signal with the smallest effective latency .
Alternatively, we can equivalently express theorem 1 as follows:
The necessary condition for is the vertex with a signal that satisfies first (see the proof of theorem 1). Given pairs of value for each and a constant value at , must also satisfy , given that the ratio of has .
2.5 Dynamics in an Absolute versus Relative Temporal Reference Frame
In a strict sense, the criterion for successful activation of a node can be described using a continuous absolute time frame of reference without the need for considering an observation time . This can be achieved in a straightforward way. Let be the start of the refractory period at the moment is activated by . The necessary condition for activation can then be restated as . reflects the time at which stops being refractory. If the node becomes refractory at and its refractory period lasts for a period , then it will stop being refractory after . The condition for activation will be met when the signal from arrives after this time—namely, after . Correspondence with the description above that makes use of a relative framework of observation time can be recovered by the relationship .
While mathematically accurate, this approach has a number of important functional and practical limitations that necessitate the consideration of the refraction ratio not from the perspective of absolute continuous time but from a relative observation or measurement time. A formulation of the model in terms of the absolute times and requires that one be able to observe the network from the start of the evolution of the dynamics—in other words, from a dynamically quiescent state. This is because one has to be able to observe the various times and for all the nodes in the network as they happen in order to compute the corresponding refraction ratios. A problem arises, however, if one needs to calculate refraction ratios for a network that has not been observed from the start of its dynamics. In this case, past times and from the moment the dynamics starts are unknowable. This is indeed the case for networks that never turn off (e.g., the brain) or in the context of machine learning frameworks that make use of this model whereby the machine is initiated and allowed to run for a period of time prior to its observation—for example, at the time when data for learning are presented to it and the network computationally interrogated. Similarly, exploring the refraction ratio of neurobiological preparations makes it impossible to know the entire dynamical history of the system. While making appropriate measurements at some may be difficult depending on the system being studied, one is at least guaranteed that computation of the winning vertices and evolution of the dynamics can in principle be computed and avoids a condition that would guarantee an inability to track the system's dynamical history—namely, needing to know unobserved measurements from the past.
A related consideration is that even if one were able to observe the dynamics from its inception (i.e., the total history of the system), it could be very computationally expensive to do so. But by referencing a relative observation or measurement time , the same calculations are based only on the current (observable) state of the dynamics, not on its past. This could save significant memory resources, since the amount of information about the system that needs to be retained is much much smaller.
3 Extensions Beyond the Basic Construction
In this section we introduce a number of natural extensions to the basic construction of the framework. We first discuss a probabilistic version of the framework, an approach for dealing with inhibitory inputs, and how the explicit contribution of the internal processing time of nodes affects the timing of subsequent output signals, which in turn affects the refraction ratio. We conclude by considering a version of the framework that accounts for a fractional contribution of summating signals. This in effect represents a geometric dynamic version of the classical perceptron. These are not mutually exclusive, and one can consider an implementation of the framework that simultaneously accommodates any or all of them, adding to the dynamical richness of the model.
3.1 Probabalistic Extension of the Framework
The construction of the framework as introduced so far is deterministic in that one individual node is capable of activating , . In other words, an activation of by is guaranteed to produce an output. This is equivalent to stating that the probability of responding to a “winning” signal from is one. However, we can extend these concepts to add a probabilistic element to the framework. For example, biological neural networks are not deterministically precise and are often subject to random fluctuations in how one neuron signals another due to thermal dynamical and subdiffusion considerations associated with molecular events such as neurotransmitter vesicles crossing the synaptic cleft or the stochastic nature of binding events on the postsynaptic membrane. Such a probabilistic extension would more accurately capture these aspects of the neurophysiology. An open question, however, is to what degree such probabilistic considerations improve the functional properties of the model in an engineering sense.
3.2 Inhibitory Inputs into
We can also extend the framework to include inhibitory inputs from in the following way. Given a “winning” vertex , , generates an output signal in turn if the input from was excitatory or does not produce an output but still becomes refractory if the input from was inhibitory. The functional result is the lack of an output from (i.e., no contribution to the activation of its downstream vertices), while preventing further activation of until ends. This is not the only way one can think of differentiating between excitatory and inhibitory inputs, but it reflects a simple approach.
3.3 Explicit Contribution of the Internal Processing Time of
In some (most) cases, the activation of a node would not result in an instantaneous output but would require a finite period of processing time associated with the node's internal dynamics prior to the generation and output of a signal. This is a function of the internal dynamic model of the node independent of the network dynamics. The details of such an internal model can be node or node class specific. We do not need to make any such distinction or have any knowledge about what the internal model is. The consequence of an internal processing time on the dynamics of the network through an analysis of our framework is a contribution to the effective latency of the signaling event reaching downstream nodes. The longer the internal processing time is, the longer the effective latency will be. This is strictly different and independent from (that is, and can evolve in parallel with) the node's effective refractory period. If this internal processing time approaches the timescale of the signaling speed and refractory period, it will affect the dynamics of the network. If, however, it is much smaller than both, it will have a negligible effect on the dynamics and can be ignored. In the limit as it approaches zero, it reflects a (near) instantaneous turn around time between when a signal that activates the node arrives and when that node outputs a signal in turn.
Before continuing, we note a potential source of confusion with regard to the node subscript notation that has to be made clear. An internal processing time is a property of node following its activation. However, it affects the latency of an outgoing signal from ; traveling along the edges, it connects to downstream nodes. In the context of signals leaving , its subscript in effect changes from to because it is now an input into the nodes it connects into. The subscript notation is of course relative to the context of the individual , node pairs under consideration. This is the mathematical equivalent of the relative usage of the terms presynaptic and postsynaptic when considering biological neuronal signaling. (A postsynaptic neuron receiving postsynaptic potentials from its presynaptic neurons becomes the presynaptic neuron once its own action potentials reach its synaptic terminals and the cell passes along the signal to the neurons it then connects to.) We attempt to be as clear and explicit as possible regarding this distinction in the description that follows.
Similar to the notation introduced previously, let be the time at which an activated node outputs a signal. Let be the period of the internal dynamic processing time by . This is the time between when a “winning” signal arrives at and when actually sends out an output at .
In the description of the framework in section 2, we implicitly assumed that , reflecting an instantaneous output at the moment that is activated. must begin anywhere between the moment of activation of when for the winning node ends, and . But this is strictly a property of the internal dynamics of . If (i.e., if there is some period of internal processing time between when is activated and when it sends an output signal at time ), there will be a lag before initiates an outgoing signal. It affects the timing of when its signaling events reach the vertices it connects to. Recall that for any vertex , we denote the time at which it initiates an outgoing signal as . This is where the switch in index notation must occur. For an activated vertex with , we write here its signal initiating time as to indicate that this particular corresponds to the activated node .
Three scenarios fully describe the range of possible effects of an internal signaling delay. First, a signal arrives at after which some amount of internal processing time given by makes a decision to output a signal and does so instantaneously following that decision. Second, a signal arrives at , which makes an instantaneous decision to output a signal, but it takes before the signal from actually goes out. The third is a combination of scenarios 1 and 2 whereby some fraction of represents the time required to make a decision and represents the time between when a decision is made and an output signal actually goes out. Note, of course, that independent of the magnitude of , the ratio . From a practical perspective, however, we do not need to distinguish among these three conditions. From the perspective of the framework, if produces a signal due to , then will effectively cause an elongation of .
If for a specific system , it would have no effect on the dynamics. For any real physical system, must always be finite and greater than zero, but we can safely ignore its effects if its timescale is much shorter than then the dynamics of the network. The computation of the refraction ratio as given does not change. But the effect on could affect the global dynamics of the network if is on the scale of .
3.4 Geometric Dynamic Perceptrons: Summation from Fractional Contributions of Multiple Nodes
In this section, we describe the summation of fractional contributions of multiple winning nodes toward the activation of . Instead of a single node activating , whether deterministically or probabilistically, we now consider a running summation of contributions from a number of adding up to a threshold value that then activates . Upon activation, becomes refractory, as previously described. Conceptually, this represents a competitive refractory model extension of the classical notion of a perceptron, whereby the metric considerations, and therefore signaling latencies, of the model play a critical role in determining node activation. These perceptrons can be thought of as having a geometric morphology or shape to account for computed latencies on the edges that represent the inputs into , similar to biological neurons (e.g., see Figure 1b). The interplay between the latencies and timing of discrete signaling events on the input edges, and the evolving refractory state of , determines the running summation toward threshold of signal contributions from arriving inputs. We assume weights and an activation function as per the standard perceptron model (see below). We also introduce a decay function that provides a memory or history for previous arriving signals, resulting in diminishing but nonzero contributions toward the summation from inputs that arrived at previous times relative to an observation time . This is how we account for a relative contribution of signals given their temporal offsets. Thus, from the perspective of the framework and refraction ratio, the computational prediction being made is not which will activate after , but what subset of will do so.
Consider a , and assume that is not refractory. As a function of time, the running summation from must reach a threshold in order for to activate at some time . Once activated, becomes refractory for a period as usual. The specific contribution from one will be the value of a weight (synaptic strength), . As is typical in a perceptron, we assume a set of weights associated with all incoming connections . In our model, the maximum value of the contributing weight occurs at the time that the signal from arrives at after the end of the relative refractory period for , . This occurs at the time (see theorem 2)—explicitly, for a time-varying weight that decays over time. Beginning at the next time step, we assume the contributing value of the weight starts to decay as a function of time: for times . In other words, there is a finite memory at future times to the arrival of a given signal, scaled to its weight value, that progressively decays over time to zero. This produces a complex and dynamic interplay between the latencies of discrete signaling events relative to each other (which in turn encodes an underlying geometry to the perceptrons and the network), the magnitude of the contribution from the respective weights once they do arrive, the kinetics of the decay of the contributing weights as a function of time, and the timing of the refractory state and recovery from the refractory state of . Excitatory versus inhibitory weights can be handled as discussed in section 3.2 in the sense that inhibitory weights have a negative impact on .
Note also that this construction differs from the probabilistic extension of the deterministic version of the framework where one individual is capable of activating . For that, we assigned a probability distribution to the likelihood of activation of given a winning vertex . Here, each contributes a component to the running summation . One could easily, however, extend the model further and construct a probabilistic fractional summation version by combining both approaches. This would add even further computational complexity to the total solution space represented by the combinatorial output of the network. However, we do not explicitly take this into account here.
We can again make use of the refraction ratios in the set to compute when with a decaying memory. Assume a nonrefractory vertex at some observation time . We consider the value of the summation of weights into for all contributing . We want to compute what subset of will result in a summation that exceeds the threshold value and at what time this will occur. Once is activated, it becomes refractory for a period . This will involve taking into account the weights , values of for , and the value of at .
Beyond the scope of this letter, we are exploring the construction and use of artificial neural networks constructed from geometric dynamic perceptrons. This work is part of the development of a fundamentally new machine learning architecture that does not necessitate statistical learning methods and requires no exposure to training data.
4 Visualizing the Dynamics with Feynman-Like Diagrams
In this section, we introduce two different versions of a diagram that provide a visual representation and summary of the dynamic model and signal summation process (see Figure 3). While it takes a bit of explaining to understand how each is constructed, they provide a compact way of visualizing the combination of processes involved and summarize a lot of information all at once. The well-known Feynman diagrams from particle physics are the inspiration for their structure, although this is immaterial for the discussion here. Note that we do not compute or show data for actual networks, since the focus of this letter is on the mathematics and theory of the model. However, the data computed by the algorithms are capable of generating these plots. Each version of the diagram summarizes the evolving temporal dynamics for a single subgraph into a vertex (see Figure 2). There would exist one such diagram (of each version) for all the vertices in the network, that is, for all . The entire global evolution of the network dynamics is then summarized by the set of all such diagrams. The intent here, however, is to introduce these diagrams as a visualization tool for the evolution of the local dynamics for a specific vertex . The difference between the two versions is that in Figure 3A, the entire causal history of is preserved for all incoming vertices , while in Figure 3B the diagram shows the combined degree or amount of summation from all the vertices into at any given moment in time. We discuss each in detail.
4.1 Visualizing the Entire Causal History of the Dynamics
Figure 3A provides a visual record of the entire causal history of for all incoming vertices as it evolves in time—in other words, the ordered causal sequence of activation events. Time is represented on the -axis, so the temporal progression of the dynamics evolves as one moves up the diagram. The initial moment of observation of the network is indicated as the horizontal dotted line at . The collection of dots indicating vectors that begin at represents a record of all signals already traveling on their respective edges of the subgraph at the moment the network is observed. The -axis represents the relative spatial positions of where traveling signals are relative to each other and the target vertex in a way we describe below.
We go through each part of the diagram in detail, with the paragraph numbers corresponding to the indicated numbering in Figure 3. For simplicity and clarity, we illustrate only excitatory inputs and do not show any inhibitory vectors in the diagram.
1. The vertical blue lines represent the state itself. In this example as drawn here, at time , is partially refractory (with period of time remaining) from a signaling activation at some time prior to . This is indicated by the length of the solid blue vector progressing forward in time. Once it stops being refractory, the solid blue vector changes to a dotted blue line, indicating is no longer refractory and capable of receiving and responding to subsequent incoming signals. Once signaling events again begin to arrive and summate, the dotted blue line changes to a squiggly blue line to show this. Eventually, once enough summation has occurred, an activation event occurs, and again becomes refractory for a period of time, switching back to a solid blue vector. Now we observe the full refractory period . Of course, the initial state of at for any particular diagram could start with any of these.
2. Each of the dots at corresponds to a signaling event that is part way along its edge. The vectors that begin at these dots end when they reach . The slope of each vector provides a measure of the remaining time it takes the signal to arrive. One can read across to the time axis from the end of the vector once it hits the blue lines. The position along the spatial -axis is relative in the sense that unlike the time axis, the origins of the vectors at each dot simply indicate how far away each signal is at from the spatial position of relative to each other; more to the left of corresponds to signals that at need to travel farther than signals farther to the right. Assuming a constant signaling speed or conduction velocity for all , as we have throughout the letter, the remaining time it takes signaling events that are already partially along their edges at the time the graph is observed at will be proportional to the distance on the edge left to travel, . The slope of each vector reflects the remaining edge distance and therefore time required for the signal to reach . Note that if the spatial axis was in real units, then the spacing between the origin of each vector would represent the actual physical distance of the signal from and assuming a constant for all signals every vector would be at 45 degrees. This is another valid way of drawing it. Here we chose to keep the spatial axis relative in arbitrary units to keep the diagram more compact and allow the slope to encode the spatial information that produces the remaining time the signals take to reach .
In the illustrative example here, we see that the first signal arrives while is still refractory and has no effect. The second signal begins a running fractional summation . The next two signals add to . The fourth arriving signal causes and fires, making it refractory again for the period .
3. Any vector that begins at a time later than represents a signal that left its origin from one of the vertices that connect into after the observation time . Note how in the example for the vector shown here at position 3 in the diagram, the signal represented by this vector can represent either one of two possible conditions: (1) it originated at a node that did not have a signal partially traversing its edge at , or (2) it corresponds to a signal from a that did have a partial signal at and fired again. If condition 2, then the signal must have originated from a vertex that corresponded to one of the first two vectors (signals) to its right at , that is, one of the two vertices closest to the spatial position of at , since its total edge length cannot be shorter than a partial edge distance of at . To be fair, it is not possible to distinguish these two conditions from the diagram as drawn without a vertex label. However, the algorithms readily keep track of this too if necessary. This diagram is able to visualize the dynamics in a way that summarizes a lot of information that can be quickly gleamed.
4. Finally, in position 4 in the diagram, we show two signals originating from the same but occurring at different times. Because we assume that is constant, the slopes for both vectors, which encode the time it takes for these signals to travel the full for a particular , will be the same.
4.2 Visualizing the Dynamics of the Summations
In Figure 3B, there is less information in the sense that the signaling events that lead to the activation of are compactified into a single vector. But the result is the ability to intuitively visualize the progression of the running fractional summation that leads to the activation of . The time axis on the -axis remains the same, but the -axis is no longer the relative spatial position of the signals from vertices that make up the subgraph. Rather, it is the degree or amount of itself. The fractional summation culminates at the vertical dotted line indicating the threshold summation value required to trigger an activation or firing of at :
1. Vectors in black show an accumulation or increase in . Here, they represent the totality of summations producing an increase in . This does not mean only the summation of excitatory inputs adding to but also counteracting inhibitory inputs that decrease or delay (depending on how inhibitory inputs are accounted for). Importantly, a component of this vector is the decay of weight contributions for both excitatory and inhibitory signals as determined by the decay equation 3.4. Black vectors that increase imply that excitatory summations outpace inhibitory summations that detract from due to a combination of the number of excitatory inputs or frequency of excitatory signals.
2. Red vectors are the same as black vectors but represent the opposite process: net inhibitory summations that are greater than excitatory summations. This then reduces the value of .
3. Through these processes, there will be a back-and-forth fluctuation of summating events that push and pull to and away from . This is the model equivalent of and directly analogous to subthreshold membrane potential fluctuations in the dendritic trees of biological neurons. In real neurons, spatial and temporal summation in dendrites attempt to reach the membrane threshold potential at the initial segment in order to trigger an action potential. As discussed at the beginning of the letter, this is in effect what our framework models and abstracts out from the neurobiology. It represents the main motivation for this work. In the example shown here at position 3 in the diagram, an inhibitory summation process that is lowering all of a sudden reverses because the summation of excitatory events overtakes it and starts driving toward .
4. Here an activation event is illustrated. reaches . A red vector must follow for at least a period corresponding to due exclusively to the decay function driving down , since will not be able to respond to inputs, excitatory or inhibitory.
5. Once ends, if the summation conditions are right, will begin to increase again. But note how the red vector can continue longer than the period dictated by if summation conditions are not favorable.
It should be clear how the multiple summating vectors in panel A that trigger an activation at correspond to the equivalent compactified perspective in panel B. The entire mathematical description introduced in the preceding sections can be visualized in these two ways. In subsequent work, we will explore the use of these Feynman-like diagrams to study how the local rules that drive the dynamics of the individual subgraphs that these diagrams represent produce the global behaviors and dynamics of the overall network.
5 Efficient Signaling
In this section, we prove a notion of optimized efficient signaling in the context of the framework and the refraction ratio. These arguments emerge naturally from a consideration of how the ratio is defined. For convenience, we limit our discussion to the deterministic version of the framework. Given an effective refractory period and effective latency along an edge , , is dependent on the for —in other words, the first discrete signal that arrives at after it stops being refractory. Intuitively, efficient signaling in the context of the framework means that there should be a temporal match between the arrival time of the signals relative to how quickly nodes can internally process signals. When a mismatch between these two considerations occurs, it can cause a breakdown in the signaling dynamics of the network. For example, we have previously shown numerically (in computational simulations) that if signaling speeds are too fast or, equivalently, if the latencies are too short, compared to the amount of time a node requires to process a signal and recover to a state in which it can respond again to another input, the network will not be able to sustain internal recurrent activity (Buibas & Silva, 2011). This reflects a form of inefficient signaling in the sense that the network requires constant external driving (energy) to sustain it.
Conversely, if is too slow or the set of too long, the network will be inefficient in a very different sense. It has the potential for faster dynamic signaling that is not being realized. Time, as a resource, is being wasted in such a network. The following discussion formalizes these concepts.
We first derive upper and lower bounds for the refraction ratio between connected vertices and . We argue that these bounds formalize a notion of optimal efficient signaling that reflect a match between the rate of discrete signal propagation (information flow) between vertices relative to the internal dynamics of the vertices.
The necessary condition for the activation of by is . By equation 2.3, which implies that . is bounded by its very construction. The absolute lower bound on implies that activation of by a , , will be achieved when (i.e., is just slightly larger than zero) and the absolute upper bound implies that . But note how can always achieve these bounds independent of for a given pair at because by equation 2.4, ; any vertex can activate independent of the absolute latency by delaying the initiation of an output signal at long enough if is too short or initiating a signal at prior to if is too long. However, by equation 2.6 and theorem 1, need only be slightly larger than in order to successfully signal (i.e., ). Because is naturally bounded by , it follows that the optimal signaling condition will be given by for values of not much smaller than zero or not much greater than zero in order to meet the condition that while avoiding compensation by . In other words, the response dynamic range for any will always be bounded by the limits of in the sense that these limits determine the temporal properties of when can actively participate in network signaling and when it cannot. Ultimately, of course, this is a function of 's internal dynamics, which in turn determines and . No value of need be much greater than for any with a directed edge into . When the condition is met, it ensures that such a is guaranteed to be able to operate over the entire response dynamic range of (i.e., all values of ). Given this condition for optimized signaling in the context of the refraction ratio, then, for the upper bound, this optimized boundary condition will occur when (when and ) because it represents the upper achievable limit for (when ) and forces the optimal condition that without compensating with .
For the lower bound, the optimal condition is given by when , since when . Forcing the condition that implies that on its own is capable of meeting the lower bound without compensation by . We can define an optimized bound as for some bounded error . If is too short, either because the path length of is too short or is too fast, this implies that given , if . To achieve the lower bound, it would require so that . To achieve the upper bound, it would require so that . If is too long, either because the path length of is too long or is too slow, this implies that given , if . When these constraints are met, it ensures that . For the lower bound, the important condition is that when . It is trivial what is, since this condition will always be met when . But for the upper bound the important condition is that when , which implies that in every case .
What this implies is that if satisfies the bounding conditions, it will always be within a range where it could efficiently “win” and activate for any value of at an observation time and time of signaling initiation by at . By efficient, we mean without forcing to compensate. The given may not of course always “win” in activating , but it is assured to be as efficient as possible—as efficient as any other node in its signaling of over all values of . If is the same for all nodes in the network, , this in effect bounds the dynamic window over which all network dynamics—that is, all temporal information (signaling) processes—should occur on that network. Explicitly, this dynamic window is given by , so there is no need or reason for any to go far beyond this dynamic window in order to satisfy the optimality condition . This is in essence the intuitive idea of using the bounds derived in theorem 3 to define optimized signaling or information flow between node pairs. Note that we must keep the explicit condition that because that forces only when . Otherwise, in the general case, any value of will result in for any value of .
We can also prove the following corollaries.
The condition for the upper bound is for (and ). Since , if , then . For a given and for a known or measurable , , since this is what the value of would have to be in order to achieve the optimal condition. For the lower bound, when given that . Thus, if or, more correctly, if , it implies that would be needed to meet the lower-bound optimality condition. The difference between and is therefore . Thus, .
If a signal characterized by a latency on the edge is able to achieve either the optimal upper bound or optimal lower bound as per theorem 3, then it is guaranteed to be able to achieve the other optimal bound.
A strict definition of an optimally efficient network follows. In every case, as a function of the effective refractory period and effective delay time along the edge , the condition for the winning vertex that achieves activation of (i.e., ) is dependent on the . When this condition is satisfied for all edges , for all node pairs by the upper- and lower-bound definitions for in theorem 3 (see equation 3.16) such that , the network is optimally efficient (i.e., . This is equivalent to requiring the condition for some arbitrarily small value of .
In this letter, we presented an intuitively simple framework that describes the competing dynamics of signaling and information flows in spatial-temporal (geometric) refactory networks derived from foundational principles of biological cellular neural signaling. These results provide a systematic explanation for how the interplay between strictly local geometric and temporal process at the scale of individual interacting nodes ultimately gives rise to the global behavior of the network. At the core of the model is the assumption that the response of each vertex is causally independent from whatever all the other vertices in the network are doing, even though collectively, they contribute to the global dynamics. The internal (to each vertex) determination of the state is not dependent on any average metric of the state of the network as a whole or on statistical probability densities associated with the frequency of occurrence of events such as in Markovian processes. This allows us to model and simulate network dynamics by keeping track of the latencies associated with signals on the edges without needing to account for or model any number of dynamic variables necessary to describe the internal models of the individual nodes that we may have little to no information about—or necessitating the computation of statistical variables from which network dynamics is then inferred, which generally precludes real-time analysis due to the need of observing or measuring sufficient data first. This is in direct contrast to most other network analysis approaches that attempt to measure or capture global properties.
The class of networks we define and investigate here can be thought of as belonging to a broader class of networks refered to as spatial-temporal and diffusion networks. These networks are of relevance to many natural, engineering, and technological systems, because the conceptual model they provide naturally maps to many real-world physical systems (Batagelj, Doreian, Ferligoj, & Kejzar, 2014; Cencetti, Clusella, & Fanelli, 2018; George & Kim, 2013; Gomez-Rodriguez, Balduzzi, & Schölkopf, 2011; Gong et al., 2018; Shanmugam, Mani, Rajan, & Joo, 2018). Examples include transportation systems, communication networks, social networks, the spread of infectious diseases (including computer viruses and malware), physical-chemical systems such as the interactions of particles in solution and diffusion, and -omics-type biological networks of molecular and genetic interactions. However, much of the literature on these classes of networks has focused on descriptive theory, definitions of concepts and notation, and various functional metrics. There are comparatively fewer applications, algorithms, and testable predictions. There is an almost exploratory quality to the existing literature. In particular, there is no work we are aware of that develops a theoretical foundation that takes advantage of canonical neurobiological processes to arrive at a set of practical algorithms, as we do here, with applicability to both neuroscience and machine learning. We have previously argued that theoretical neuroscience, while challenging, offers an opportunity for a deeper understanding than purely numerical or simulation-based computational neuroscience models (Silva, 2011).
As an example of a real-world application of the ideas and framework we discuss here, of particular interest to us is the signaling dynamics responsible for computation in biological neural networks across scales of organization (e.g., dendritic trees, neural circuits, or the interaction between different brain regions) constrained by the underlying structure—in other words, the structure-function relationships responsible for neural dynamics. We recently showed that basket cell neurons in the cortex are putatively designed to optimize for the refraction ratio. They seem to be designed to optimize the ratio between the refractory period of the membrane and action potential latencies along the axon between the initial segment (where action potentials begin) and the synaptic terminals (where they end; Puppo et al., 2018). Dynamic signaling on branching axons is critical for rapid and efficient communication between neurons in the brain. Efficient signaling in axon arbors depends on a trade-off between the time it takes action potentials to reach synaptic terminals (temporal cost) and the amount of cellular material associated with the wiring path length of the neurons' morphology (material cost). However, where the balance between structural and dynamical considerations for achieving signaling efficiency is, and the design principle that neurons optimize to preserve, this balance has remained elusive. We took advantage of the theoretical prediction of theorem 3 and went looking to see if biological neurons displayed optimized signaling in the context predicted by the refraction ratio. One interpretation of our results is that the convoluted paths taken by axons reflect a design compensation by neurons to slow down, signaling latencies in order to optimize the refraction ratio. This was not a serendipitous discovery. We went looking to see if neurons actually optimize the refraction ratio because the mathematics pointed us in that direction.
I am grateful to Fang Chung (Department of Mathematics, UC San Diego) for her input and feedback when this work was in its earliest stages and to Vivek George (Department of Bioengineering, UC San Diego) for many fruitful discussions. I sincerely acknowledge the effort of the two anonymous reviewers. This was not an easy manuscript to review. Their very thoughtful feedback has strengthened the final work product. This work was supported by grants 63795EGII and N00014-15-1-2779 from the Army Research Office (ARO), U.S. Department of Defense, and in part from unrestricted funds to the Center for Engineered Natural Intelligence.