Models of cortical neuronal circuits commonly depend on inhibitory feedback to control gain, provide signal normalization, and selectively amplify signals using winner-take-all (WTA) dynamics. Such models generally assume that excitatory and inhibitory neurons are able to interact easily because their axons and dendrites are colocalized in the same small volume. However, quantitative neuroanatomical studies of the dimensions of axonal and dendritic trees of neurons in the neocortex show that this colocalization assumption is not valid. In this letter, we describe a simple modification to the WTA circuit design that permits the effects of distributed inhibitory neurons to be coupled through synchronization, and so allows a single WTA to be distributed widely in cortical space, well beyond the arborization of any single inhibitory neuron and even across different cortical areas. We prove by nonlinear contraction analysis and demonstrate by simulation that distributed WTA subsystems combined by such inhibitory synchrony are inherently stable. We show analytically that synchronization is substantially faster than winner selection. This circuit mechanism allows networks of independent WTAs to fully or partially compete with other.
Many models of neuronal computation involve the interaction of a population of excitatory neurons whose outputs drive inhibitory neuron(s), which in turn provide global negative feedback to the excitatory pool (Amari & Arbib, 1977; Douglas, Koch, Mahowald, Martin, & Suarez, 1995; Hahnloser, Sarpeshkar, Mahowald, Douglas, & Seung, 2000; Yuille & Geiger, 2003; Maass, 2000; Hertz, Krogh, & Palmer, 1991; Rabinovich et al., 2000; Rutishauser, Douglas, & Slotine, 2011; Coultrip, Granger, & Lynch, 1992). Practical implementation of such circuits in biological neural circuits depends on colocalization of the excitatory and inhibitory neurons, an assumption that studies of the extent of axonal and dendritic trees of neurons in the neocortex show is not valid (Katzel, Zemelman, Buetfering, Wolfel, & Miesenbock, 2011; Binzegger, Douglas, & Martin, 2004; Shepherd, Stepanyants, Bureau, Chklovskii, & Svoboda, 2005; Douglas & Martin, 2004). First, a substantial fraction of the axonal arborization of a typical excitatory “spiny” pyramidal neuron extends well beyond the range of the arborization of a typical “smooth” inhibitory neuron, particularly in the populous superficial layers of the neocortex (Yabuta & Callaway, 1998; Binzegger et al., 2004). This spatial arrangement means that excitatory effects can propagate well outside the range of the negative feedback provided by a single inhibitory neuron. Second, the horizontally disposed basket type of inhibitory neuron, a prime candidate for performing normalization, makes multiple synaptic contacts with its excitatory targets, so that even within the range of its axonal arborization, not all members of an excitatory population can be covered by its effect. This connection pattern means that excitatory neurons within some local population must either be partitioned functionally or multiple smooth cells must cooperate to cover the entire population of excitatory cells.
In previous publications, we have shown how winner-take-all (WTA) circuits composed of a small population of excitatory neurons and a single inhibitory neuron can be combined to construct supercircuits that exhibit finite state machine (FSM)–like behavior (Rutishauser & Douglas, 2009; Neftci, Chicca, Indiveri, Cook, & Douglas, 2010). The supercircuits made use of sparse excitatory cross-connections between WTA modules to express the required states of the FSM. These excitatory connections can extend well outside the range of the local WTA connections, and so are consistent with the observed long-range lateral excitatory connections referred to above. On the other hand, we have not previously confronted the question of whether the WTA is necessarily localized to the extent of the smooth-cell arborization, or whether the WTA itself can be well distributed in space within or between cortical lamina, or even between cortical areas. In this letter, we describe a simple modification to the WTA circuit design that couples the effects of distributed inhibitory neurons through synchronization, and so permits a WTA to be widely distributed in cortical space, well beyond the range of the axonal arborization of any single inhibitory neuron, and even across cortical areas. We also demonstrate that such a distributed WTA is inherently stable in its operation.
We have considered a number of circuits that could be used to distribute spatially the WTA behavior (see Figure 1). However, we will describe and analyze only the circuit shown in Figure 2, which we consider to be the most elegant of the distributive mechanisms (notice the similarity to Figure 1B). The key insight is the following: under normal operating conditions, all the participating distributed inhibitory neurons should receive the same summed excitatory input. We achieve this by interposing an excitatory neuron in the negative feedback loop from the excitatory population to its local inhibitory neuron. Instead of the local inhibitory neuron summing over its excitatory population, the interposed neuron performs the summing and passes its result to the inhibitory neuron. This result is also copied to the more distant inhibitory neurons in the spatially distributed WTA. In this way, the inhibitory neuron of each sub-WTA sums over the projections from the interposed excitatory neurons of all other sub-WTAs, including its own one. Thus, each inhibitory neuron is able to provide feedback inhibition to its local sub-WTA that is proportional to the total excitation provided by all excitatory neurons participating in the entire distributed WTA. We will show that functionally, this amounts to a form of synchrony among all the inhibitory units.
2.1. Connectivity and Dynamics: Single WTA.
All of the circuits of Figure 1 can achieve a distributed WTA by merging several independent WTAs, but we consider the circuit shown in Figure 2B to be most feasible, and so our analysis focuses on this one. However, similar reasoning could be applied to all the variants shown. Note that our chosen circuit is similar to that of Figure 1B, but it has a more realistic connectivity pattern in that the summed excitatory activity is projected onto only a single unit, which requires less wiring specificity than Figure 1B.
2.2. Connectivity and Dynamics: Coupled WTA.
2.3. Stability Analysis.
The stability analysis, using nonlinear contraction analysis (see the appendix; Lohmiller & Slotine, 1998, 2000; Slotine, 2003; Wang & Slotine, 2005), consists of three steps: (1) demonstrate the contraction of a single WTA, (2) merge two WTAs by demonstrating that inhibitory units synchronize, and (3) demonstrate the contraction of the combined WTAs. We have previously shown how contraction analysis can be applied to reasoning over the stability and functionality of WTA circuits (Rutishauser et al., 2011). Here, we apply and extend the same methods to this new circuit.
Contraction analysis is based on the Jacobians of the system. For purposes of analysis but without loss of generality, we base this section on a reduced system with only one possible winner for each WTA, as shown in Figure 2C.
The activation function f(x)=max(x, 0) is not continuously differentiable, but it is continuous in both space and time, so that contraction results can still be applied directly (Lohmiller & Slotine, 2000). Furthermore, the activation function is piecewise linear with a derivative of either 0 or 1. We exploit this property by inserting dummy terms lj, which can be either 0 or 1 according to the derivative of f(x): . In this case, all units are active, and thus l1=l2=l3=1.
2.3.1. Synchronizing Two WTAs.
Next, we show that connecting two WTAs in the manner illustrated in Figure 2C results in synchronization of the two inhibitory units, which in turn leads to the two WTAs merging into a single WTA. Note that by synchronization, we mean that the two variables have the same trajectory, or more generally that their difference is constant (in contrast to other meanings of synchronization, such as in population coding). The approach is to show that adding excitatory connections of sufficient strength will lead to the activity of the two inhibitory units x2 and y2 approaching a constant difference.
2.3.2. Stability of Pairwise Combinations of WTAs.
The final step of the stability analysis is a condition for the coupling strength such that the coupled system as shown in Figure 2C is contracting. The reasoning in this section assumes that the individual subsystems are contracting (as shown above).
2.3.3. Summary of Boundary Conditions.
2.4. Speed of Winner Selection.
How quickly will a system select a winner? For a single WTA, this question is answered by how quickly a single system contracts toward a winner and for a coupled system how quickly the two systems synchronize. One of the key advantages of contraction analysis is that the rate of contraction, and in this case the rate of synchronization, can be calculated explicitly. We will express the contraction and synchronization rate in terms of the time constants and its inverse, the decay constant. refers to the mean lifetime of exponential decay . is the decay constant. Both the contraction and the synchronization rate are expressed in the form of a decay constant . For example, the contraction rate of a system of the form is equivalent to .
Physiologically, the time constants in our system are experimentally determined membrane time constants that are typically in the range of 10 to 50 ms (Koch, 1998; McCormick, Connors, Lighthall, & Prince, 1985; Koch, Rapp, & Segev, 1996; Brown, Fricke, & Perkel, 1981). For simplicity, we assume that all excitatory and inhibitory units have the same, but different, time constants and , respectively. While the exact values depend on the cell type and state of the neurons, it is generally the case that due to the smaller cell bodies of inhibitory neurons (McCormick et al., 1985; Koch, 1998).
The bounds, equation 2.19, were calculated assuming equal time constants for all units. However, the same calculations yield to very similar bounds when assuming different time constants for inhibitory and excitatory units (Rutishauser et al., 2011). In this case, the ratio of the time constants becomes an additional parameter for the parameter bounds.
2.4.1. Speed of Synchronization.
The synchronization rate is equivalent to the contraction rate of the system defined in equation 2.13 (Pham & Slotine, 2007), which is the absolute value of the maximal eigenvalue of the Hermitian part of . Here, the original is replaced by the diagonal matrix R, with the appropriate terms on the diagonal.2 The matrix is an orthonormal version of V as defined in equation 2.14, which here is simply .
In general for , the sync rate is . For example, for , this reduces to . Again, for , the sync rate is approximately . In conclusion, the sync rate is thus approximately equal to the contraction rate of the inhibitory units. Thus, synchronization occurs very quickly (20–50 ms for typical membrane time constants).
2.4.2. Speed of Contraction.
The speed of selecting a winner (the contraction rate) for a single WTA can similarly be calculated based on the absolute value of the maximal eigenvalue of the Hermitian part of (see equation 2.7).
2.5. Coupling More Than Two WTAs.
So far we have shown how two different WTAs compete with each other after their inhibitory neurons are coupled. Similarly, more than two WTAs can compete with each other by all-to-all coupling of the inhibitory units, that is, every WTA is connected with two connections from and to every other WTA. Thus, the wiring complexity of this system scales as O(M2), where M is the number of WTAs in the system (note that M is not the number of units but the number of WTAs). Notice also that the all-to-all coupling concerns only the sparse long-range excitatory connections, not the internal connectivity of the WTAs themselves.
The same principle can be used to embed hierarchies or sequences of competition. Thus, in a network of such WTAs, some WTAs could be in direct competition with each other, while others are not. Thus, for example, in a network of three WTAs A, B, and C relationships such as A competes with B and B competes with C are possible. In this case, A does not directly compete with C. So if A has a winner, C can also have a winner. If B has a winner, however, neither B nor C can have a winner (see Figures 4D–4F for a demonstration).
Regardless of how many WTAs are combined and whether all compete with all or more selectively, the stability of the aggregated system is guaranteed if the individual subsystems are stable and the coupling strengths observe the derived bounds. While in themselves combinations of stable modules have no reason to be stable, certain combinations (such as the one we use) of contracting systems are guaranteed to be stable (Slotine & Lohmiller, 2001). This is a key benefit of contraction analysis for the analysis of neural circuits.
2.6. Numerical Simulations.
We simulated several cases of the network to illustrate its qualitative behavior. We used Euler integration with s. The analytically derived bounds offer a wide range of parameters for which stability as well as function is guaranteed. For the simulations, we chose parameters that verify all bounds discussed.
First, we explored a simple system consisting of two WTAs with two possible winners each (see Figure 3). Parameters were , and T=0. We found that any of the four possible winners can compete with each other regardless of whether they reside on the first or second WTA (Figures 3B–3D show an example). The inhibitory units quickly synchronized (see Figure 3C) their activity and reached the same steady-state amplitude (because ).3
Second, we simulated a system with three WTAs using the same parameters (see Figure 4). For all-to-all coupling, all three WTAs directly compete with each other (see Figures 4A and 4B), that is, there can be only one winner across the entire system. Again, the inhibitory units all synchronize quickly during and after convergence (see Figure 4C). We also simulated the same system with more selective connectivity, eliminating competition between WTAs 1 and 3 (see Figure 4D). This arrangement allows either one winner if it is on WTA 2, or two winners if they are on WTAs 1 and 3. If the maximal activity is not on WTA 2, then the network permits two winning states. Otherwise, if the maximal input is on WTA 2, only one winner is permitted (see Figure 4E for an illustration). This configuration allows partial competition.
Neural circuits commonly depend on negative feedback loops. Such recurrent inhibition is a crucial element of microcircuits from a wide range of species and brain structures (Shepherd & Grillner, 2010) and enables populations of neurons to compute nonlinear operations such as competition, decision making, gain control, filtering, and normalization. However, when biologically realistic versions of such circuits are considered, additional factors such as wiring length, specificity, and complexity become pertinent. Here, we are principally concerned with the superficial layers of neocortex where the average distance of intracortical inhibitory connections is typically much shorter than the excitatory connections (Bock et al., 2011; Binzegger et al., 2004; Perin, Berger, & Markram, 2011; Katzel et al., 2011; Adesnik & Scanziani, 2010). In contrast, in invertebrates, an inhibitory neuron has been identified that receives input from and projects back to all Kenyon cells (which are excitatory) (Papadopoulou, Cassenaer, Nowotny, & Laurent, 2011). This neuron has been demonstrated to perform response normalization, making this system a direct experimental demonstration of competition through shared inhibition. No such system has yet been identified in the cortex.
The number of excitatory neurons that can be contacted by an inhibitory neuron thus poses a limit on how many excitatory neurons can compete directly with one another (in terms of numbers and distance). Other models, such as those based on Mexican hat–type inhibitory surrounds (Hertz et al., 1991; Willshaw & von der Malsburg, 1976; Soltani & Koch, 2010), even require that inhibitory connectivity be longer range than the excitatory. These anatomical constraints have been used to argue that models such as the WTA are biologically unrealistic and therefore of limited use.
We have demonstrated here, by theoretical analysis and simulation, that it is possible to extend such circuits by merging several independent circuits functionally through synchronization of their inhibitory interneurons. This extension allows the construction of large, spatially distributed circuits that are composed of small pools of excitatory units that share an inhibitory neuron. We have applied and proved by nonlinear contraction analysis that systems combined in this manner are inherently stable and that arbitrary aggregation by inhibitory synchrony of such subsystems results in a stable system. This composition of subcircuits removes the limits on maximal circuit size imposed by anatomical wiring constraints on inhibitory connectivity because the synchrony between local inhibitory neurons is achieved entirely by excitatory connectivity, which can possibly be long range, permitting competition between excitatory units that are separated by long distances, for example, in different cortical areas. We show that the time necessary to achieve sychronization is much shorter than the time required to select a winner. Thus, synchronization is faster than winner selection, which can thus proceed robustly across long-range connections that enforce synchronization. Further, selective synchronization among some WTAs but not others allows partial competition between some but not other WTAs (see Figure 4). The strength of these long-range connections could be modulated dynamically to enable or disable various competitions between two populations conditional on some other brain state. This modulation could be implemented by a state-dependent routing mechanism (Rutishauser & Douglas, 2009).
There are several possibilities for mapping the abstract units in our model to real physiological neurons. Our units are mean-rate approximations of a small group of neurons. In terms of intracortical inhibition, these would lie anatomically close to each other within superficial layers of neocortex. Since such inhibitory connectivity would have only limited reach, each inhibitory subunit can enforce competition only across a limited number of close-by excitatory units. Competition between different areas is made possible by synchronizing remote populations by long-range excitatory mechanisms in the way we propose. Direct long-range inhibition, on the other hand, is unlikely both intracortically and subcortically, since all known connections from the thalamus and basal ganglia to cortex are excitatory. Networks such as the LEGION network (Wang & Terman, 1995) assume global inhibitory input to all excitatory units in the network, which, for the reasons we discuss, is unlikely in the case of cortex. It would, however, be possible to implement a feasible version of the global inhibitory input by synchronizing many local inhibitory neurons using the mechanism we describe, resulting in an anatomically realistic version of the LEGION network.
Functionally, the model presented here makes several testable predictions. Consider a sensory area with clearly defined features as possible winners, such as orientations. The model predicts that the inhibitory units would not be tuned to these features, particularly if the number of possible winners is large. This is because the connectivity to the inhibitory units is not feature specific. Experimental studies indicate that this is indeed the case: units that functionally represent different tuning project to the same inhibitory unit, resulting in untuned inhibitory activity (Bock et al., 2011; Fino & Yuste, 2011; Kerlin, Andermann, Berezovskii, & Reid, 2010; Kuhlman, Tring, & Trachtenberg, 2011; Hofer et al., 2011). Second, this model predicts that inhibitory activity between two different areas or parts of the same area can either be highly synchronous or completely decoupled depending on whether the two are competing or functioning independently. This thus predicts that synchrony of inhibitory units should be affected by manipulations that manipulate competition, such as top-down attention.
Our model suggests that synchronized populations of inhibitory neurons are crucial for enforcing competition across several subpopulations of excitatory neurons. It further suggests that the larger the number and spatial distribution of such synchronized inhibitory units, the larger the number of units that compete with each other. Experimentally, synchronized modulation of inhibitory neurons is a common phenomenon that is believed to generate the prominent gamma rhythm triggered by sensory stimulation in many areas (Fries, Nikoli, & Singer, 2007; Whittington, Traub, & Jefferys, 1995; Traub, Whittington, Stanford, & Jefferys, 1996). Recent experiments have used stimulation of inhibitory neurons (Cardin et al., 2009; Sohal, Zhang, Yizhar, & Deisseroth, 2009; Szucs, Huerta, Rabinovich, & Selverston, 2009) to increase or decrease their synchronization with direct observable effects on nearby excitatory neurons such as, for example, increased or decreased amplitude and precision of evoked responses relative to how strongly the inhibitory neurons were synchronizing. Note that our proposal for this function of inhibitory synchrony is distinct and independent from the proposal that gamma-band synchrony serves to increase readout efficacy by making spikes arrive coincidentally from a large number of distributed sources (Tiesinga, Fellous, & Sejnowski, 2008; Singer & Gray, 1995). Here, we propose that an additional function of such synchrony is to allow select populations of excitatory neurons to compete with each other because each receives inhibition at the same time.
Appendix: Contraction Analysis
This appendix provides a short summary of contraction analysis. We have previously published the detailed methods of applying contraction theory to WTA circuits (Rutishauser et al., 2011). Essentially, a nonlinear time-varying dynamic system will be called contracting if arbitrary initial conditions or temporary disturbances are forgotten exponentially fast, that is, if trajectories of the perturbed system return to their unperturbed behavior with an exponential convergence rate. A relatively simple algebraic condition can be given for this stability-like property to be verified, and this property is preserved through basic system combinations and aggregations.
Global exponential convergence and stability are guaranteed.
Convergence rates can be explicitly computed as eigenvalues of well-defined Hermitian matrices.
Combinations and aggregations of contracting systems are also contracting.
Robustness to variations in dynamics can be easily quantified.
Before stating the main contraction theorem, recall first the following properties: The symmetric part of a matrix A is . A complex square matrix A is Hermitian if , where T denotes matrix transposition and complex conjugation. The Hermitian part AH of any complex square matrix A is the Hermitian matrix . All eigenvalues of a Hermitian matrix are real numbers. A Hermitian matrix A is said to be positive definite if all its eigenvalues are strictly positive. This condition implies in turn that for any nonzero real or complex vector, x, . A Hermitian matrix A is called negative definite if −A is positive definite.
A Hermitian matrix A(x, t) dependent on state or time will be called uniformly positive definite if there exists a strictly positive constant such that for all states x and all , the eigenvalues of A(x, t) remain larger than that constant. A similar definition holds for uniform negative definiteness.
In the linear time-invariant case, a system is globally contracting if and only if it is strictly stable, and F can be chosen as a normal Jordan form of the system, with a real matrix defining the coordinate transformation to that form (Lohmiller & Slotine, 1998). Alternatively, if the system is diagonalizable, F can be chosen as the diagonal form of the system, with a complex matrix diagonalizing the system. In that case, FH is a diagonal matrix composed of the real parts of the eigenvalues of the original system matrix.
The variable permutation is equivalent to a transformation of J1 by the permutation matrix : with .
For the example of JC (see equation 2.11), the diagonal terms are .
If , it can be verified that x3(t)=y4(t) for all T>0 if the initial values at T=0 are equal. Thus, the two inhibitory neurons become exactly equivalent in this special case.