Toward a Biomimetic Neural Circuit Model of Sensory-Motor Processing

What does it mean to understand how the brain works? I subscribe to Feynman's view: “What I cannot create I do not understand.” When I create a biologically realistic model, then I can say that I understand how a neural system works. Yet computational models of the brain take many forms, meaning that I need to specify the level of biological realism that I expect. Of course, the model must reproduce the input-output relationship of the brain system on which it is based. But to reveal how the brain works, the model will need to have circuits with realistic architecture and model neurons that produce realistic firing rates during behavior. Further, it should reproduce both the first- and second-order statistics of neural firing: mean and variance of responses across trials, trial-by-trial “noise” correlations between neurons, and trial-by-trial “neuron-behavior” correlations between the variations in spiking and behavioral output.

The sensory-motor system I have studied for the past half century offers great promise for developing the kind of biomimetic model that I describe in the previous paragraph. Smooth pursuit eye movements provide a quantifiable sensory-motor behavior that is amenable to study in nonhuman primates (Fuchs, 1967). Pursuit is driven by visual inputs, namely, by the motion of a small spot or patch of dots, and it generates smooth eye movements that take the eye in the direction of target motion at very close to target speed. It is one of the few voluntary movements where the link between sensation and action is both direct and continuously observable. Among behavioral systems, pursuit is unusually amenable to quantitative examination because we can map the input-output relationship via different visual motion stimuli and uncover situations where the baseline input-output relationship must be modulated by internal processes. To analyze the behavior, we have devised many behavioral and stimulus regimes. By going beyond the simple and obvious strategy of simply having a monkey track a moving spot, we have revealed multiple components of pursuit behavior and mapped many of those components to specific regions of the pursuit circuit. It is rare, in monkeys and rodents, to be able to characterize fully an interesting and complex input-output relationship while also understanding how it is modulated by internal parameters.

The neural circuit for pursuit is largely known, though it is probably more complicated and convoluted than we think, and important gaps still exist. Neural recordings are available from many, but not all, nodes of the pursuit circuit. Recordings were made during pursuit, and to some degree during the more elaborated set of behavioral and stimulus regimes we have developed. In some key areas, the recordings include both first- and second-order statistics of neural firing, and therefore they provide the substrate for a biologically realistic model of the neural pursuit system. Thus, smooth pursuit eye movements afford a nearly unique opportunity to understand the neural basis of a voluntary behavior and how it is created millisecond by millisecond by sensory-motor pathways at the full circuit level.

Computational modeling has a long tradition in smooth pursuit eye movements (Robinson, Gordon, & Gordon, 1986; Krauzlis & Lisberger, 1989; Ringach, 1995; Churchland & Lisberger, 2001a; Orban de Xivry, Coppe, Blohm, & Lefevre, 2013; Egger & Lisberger, 2022). However, most models have been designed to account for behavioral phenomena without regard for the architecture of signal processing within the essential circuit. Also, most models have been designed to account for a single feature of pursuit or at most a small set of features. My contributions have been as guilty of wearing these blinders as have those from everyone else. In the meantime, knowledge has expanded about pursuit behavior, the computational components of the pursuit system, and their neural correlates (Lisberger, 2015). The evolution of knowledge creates the possibility to start on a path toward a computational neural circuit model of how one sensory-motor system works in a behaving animal.

Here, I explain the evolution of knowledge and models of smooth pursuit eye movements in the time from our contribution in the first issue of Neural Computation (Krauzlis & Lisberger, 1989) to the present. I focus unabashedly on the research from my own laboratory to explain how our knowledge has evolved over 33 years. At the end of the review, I outline what I see as the next steps in computational models of smooth pursuit eye movements, including an explanation of how the predictions of future models might be tested through application of modern molecular approaches and recording with multicontact probes in behaving monkeys. Finally, I defend the choice to pursue this particular goal in monkeys, even in the face of remarkable technological advances for neuroscience research on rodents.

Our model deviated qualitatively from previous models of pursuit, which had focused on image velocity as input to the system and had placed most of the dynamics of pursuit in the motor system (Robinson et al., 1986). In our data, we found that the actual time-varying trajectory of pursuit eye velocity has two features that are not expected of a model based solely on image velocity. First, pursuit eye velocity settles at target velocity with little or no overshoot (see Figures 1A and 1C, black traces), while the latencies in the system mean that a model based on image velocity (see Figure 1C, red trace) shows considerable overshoot. Second, pursuit eye velocity sometimes oscillates at a fairly high frequency during steady-state tracking (see Figure 1C, black trace), while the image-velocity model predicts oscillations at a much lower frequency (see Figure 1C, green trace). The theoretical prediction is that an image velocity model should oscillate with a period that is four times the visual delay of 100 ms in our data (i.e., 2.5 Hz). In our data (see Figure 1C, red trace), steady-state pursuit oscillated at a frequency closer to 5 Hz (Goldreich, Krauzlis, & Lisberger, 1992).

We reproduced the actual trajectory of eye velocity during pursuit with an image-motion model (see Figure 1B) that includes much expanded visual motion processing. Here, visual motion processing is represented as the sum of (1) a traditional image velocity pathway, (2) a pathway that is sensitive only to the transient at the onset of motion, and (3) a pathway that is sensitive to smooth image acceleration. The motor system is represented by a positive feedback pathway through gain element “g2” that sustains eye velocity during perfect tracking in the absence of image motion and renders image motion signals as commands for changes in eye velocity, that is, eye acceleration (Morris & Lisberger, 1987). The full image-motion model of Figure 1B predicts eye velocity (see Figure 1C, red trace) that nicely reproduces the two main features of actual eye velocity: (1) little overshoot and (2) oscillations of eye velocity at 5 Hz. The image acceleration pathway is essential for getting the correct frequency of oscillation, and a signal related to image deceleration (see the red trace in Figure 1A) is needed to prevent eye velocity overshoot after the initiation of pursuit.

It is possible to account for much of the trajectory of pursuit eye velocity with models that place the dynamics of the system in the motor pathways and rely on simple image velocity inputs to drive the model (Robinson et al., 1986; Ringach, 1995). However, these motor-dynamics models fail under a simple experimental manipulation that inserts delays of 20 to 100 ms into visual feedback (Goldreich et al., 1992). In experiments, the frequency of oscillation in steady-state tracking decreases gracefully as a function of visual delay. The same is true in image-motion models, but motor-dynamics models make more complicated and biologically unrealistic predictions (Churchland & Lisberger, 2001a). Thus, some of the dynamics of pursuit are caused by the properties of the full sensory-motor loop. Internal recurrent feedback loops drive other aspects of the dynamics, namely, the maintenance of accurate tracking in the absence of image motion across the retina during steady-state tracking. Perhaps this division of the labor of dynamics provides a simpler model system that offers lessons for the discussion for reaching movements of whether the dynamics in the motor cortex arise from internal recurrent circuits (Shenoy, Sahani, & Churchland, 2013) or from external sensory-motor loops (Kalidindi et al., 2021).

Models suggest experiments. The successful predictions of eye velocity from the image-motion model in Figure 1B inspired Tony Movshon and me to embark on extensive recordings from extrastriate visual area MT. Our goal was to look for neurophysiological evidence of the motion-onset or image-acceleration pathways in the image-motion model. Both lesion and microstimulation experiments show that MT provides important visual motion inputs that drive the initiation of pursuit, but not for steady-state tracking. Neurons in MT are selectively sensitive to moving stimuli and tuned for both the direction and speed of motion. Thus, we knew that we would find a representation of the model's image velocity pathway in MT. We recorded from MT in anesthetized monkeys so that we could study neurons for hours using an extensive set of target motions, without the confound of the monkey tracking the target and changing the visual stimulus.

We discovered correlates of the motion-onset and smooth-image-acceleration pathways, but we did not find the image deceleration signal needed to prevent overshoot of eye velocity after the initiation of pursuit (Lisberger & Movshon, 1999). We used target motions that ramped from zero to a fixed speed and then back down to zero so that we could compare the responses of MT neurons to the same set of target speeds but during acceleration versus deceleration. Many neurons showed impressive differences in firing rate for acceleration versus deceleration (see Figure 1D), implying that image acceleration and deceleration might be represented in the MT population. Further, the transient firing of MT neurons at the onset of stimulus motion provided a clear correlate of the motion onset pathway in the image-motion model. In fact, different neurons showed different degrees of acceleration-deceleration asymmetry (ADA; see Figure 1E). We were able to capitalize on the diversity of acceleration-deceleration asymmetries to recover stimulus acceleration. We did so by weighting each neuron in the population by its ADA. A parallel computation recovered stimulus speed by weighting each neuron by its preferred speed (see Figure 1F). However, we were not able to recover image deceleration. Thus, our analysis of the signals emanating from MT did not seem to support fully the need of the image-motion model for a visual signal related to image deceleration, an issue that remains unresolved to this day.

The next step in creating a model of pursuit came from the realization that one component of the system controls the strength, or gain, of the transmission of visual signals to the motor system. Others had suggested that pursuit needed to be turned on when a target moved based on the conclusions that fixation and pursuit probably are different subsystems and that fixation probably is not pursuit of zero target speed (Robinson, 1965; Luebke & Robinson, 1988). We codified the process of switching pursuit “on” experimentally and showed that the underlying process was a volume control, not a switch (Schwartz & Lisberger, 1994).

The smooth eye movement region of the frontal eye fields (FEF_SEM) plays a major role in controlling the gain of visual-motor transmission. On a hunch, we introduced microelectrodes into FEF_SEM and performed microstimulation at sites where the neurons discharged vigorously in relation to smooth pursuit eye movements (Tanaka & Lisberger, 2001). At suprathreshold intensities, stimulation evokes a smooth eye movement. At much lower intensities, stimulation increases the size of the response to a perturbation of target motion during fixation, while evoking only a small smooth eye movement. Support for this assertion appears in Figure 2F.

Presentation of a perturbation of target motion during fixation causes a small smooth eye velocity (dashed red trace). Stimulation of FEF_SEM alone also causes a small smooth eye velocity (black dashed trace). Presentation of the same perturbation of target motion during stimulation of FEF_SEM causes a much larger eye movement (solid black trace). Subtraction of the eye movement induced by microstimulation alone from that induced by a perturbation during microstimulation leaves a response to the perturbation that is much larger during stimulation (solid red trace) than without stimulation (dashed red trace). Thus, the output from FEF_SEM controls the gain of visual-motor transmission.

The discovery of gain control led to the development of two-pathway models (see Figure 2A). Visual motion inputs arise in the responses of the population of neurons in MT. One pathway provides visual drive for pursuit and the other pathway, through FEF_SEM, sets the gain of visual-motor transmission. We will see later that the two-pathway configuration has two major virtues: it fits with the biology of two parallel pathways from MT to the motor system, and it allows the models to account for a lot of behavioral data based on visual inputs from realistic models of MT population responses.

The demonstration of gain control provides an answer for one paradox in the effects of stimulus contrast and target speed on the initiation of pursuit. It explains how the eye velocity at the initiation of pursuit can increase as a function of target speed yet decrease when the contrast of the visual stimulus is reduced. The paradox arises because of experimental observations on how the response of the population of MT neurons changes as a function of target speed and contrast, discussed below.

Here, S is the estimate of target speed, and $RiMT$ and $Sipref$ are the response and preferred speed of the ith MT neuron. The constant $ɛ$ confers noise immunity at low response amplitudes and renders the estimate of target speed S somewhat sensitive to the amplitude of the population response.

Vector averaging (i.e., equation 3.1) accounts for the effect of target speed on the initiation of pursuit (Gardner, Tokiyama, & Lisberger, 2004). To visualize the MT population response, we assemble the responses of all neurons in the population by plotting each neuron's response as a function of its preferred speed. In Figure 2C, for example, the open and filled circles show the responses of the population of six neurons for target speeds of 8 and 24 deg/s, taken from the solid speed tuning curves in Figure 2G by measuring firing rate at the vertical black and red dashed lines. Higher target speed creates a population response that peaks at higher values of preferred speed but retains the same amplitude. Vector averaging finds the preferred speed at the peak of the population response, and so it estimates the correct target speed, as does the initiation of pursuit.

Vector averaging needs gain control to account for the effect of target contrast on the initiation of pursuit. Lower-contrast targets produce lower speeds at the initiation of pursuit. Yet recordings from MT (Krekelberg, van Wezel, & Albright, 2006; J. Yang & S. G. Lisberger, unpublished data) indicate that the population response in MT shifts toward higher preferred speeds when target contrast is decreased (see Figure 2C, open triangles versus open circles), even though the amplitudes of the responses decrease for all neurons (see Figures 2B, 2C, and 2G). We account for the decrease in eye velocity in the initiation of pursuit with the suggestion that reduced contrast creates a less reliable visual motion signal in MT because it decreases the amplitude (and likely the signal-to-noise ratio) of the population response (Darlington, Tokiyama, & Lisberger, 2017). FEF_SEM interprets the less reliable motion signal as a caution that visual-motor gain should be low. Although vector averaging estimates a higher target speed for lower contrast targets, visual-motor gain counteracts that effect. As a result, the eye velocity at the initiation of pursuit decreases as a function of target contrast even as the population response in MT moves toward higher preferred speeds.

We account for the effect of dot coherence on the initiation of pursuit in the model of Figure 3C by adding the gain element G₁ to the image motion model from Figure 1B. G₁ is the instantiation in this model of the gain control provided by FEF_SEM in Figure 2A; the model places G₁ under the control of dot coherence (i.e., motion reliability).

We account for the effect of dot coherence on steady-state tracking by allowing it to control the value of G₂ in the model of Figure 3C. G₂ lies within a positive feedback pathway that uses a corollary discharge of eye velocity as part of the eye movement command and thereby sustains perfect tracking in the absence of image motion. Our simple models are based on neural recordings (Lisberger & Fuchs, 1978; Stone & Lisberger, 1990), showing that the floccular complex of the cerebellum is part of a positive feedback pathway that receives a corollary discharge of eye velocity from time t − Δt and uses the corollary discharge to control eye velocity at time t. Thus, nonvisual signals control the quality of steady-state tracking and keep the eyes moving smoothly at target speed even in the absence of image motion; image motion causes changes in eye velocity to correct any tracking errors. Simulation of the model in Figure 3C shows that it reproduces both of the effects of reduced dot coherence only if we allow modulation of both G₁ and G₂ by dot coherence (see Figure 3D). Modulation of G₁ only (see Figure 3E) or G₂ only (see Figure 3F) fails to reproduce the biological data (Behling & Lisberger, 2020).

In biological terms, we understand the effect of reduced dot coherence as an effect on the internal reliability, or signal-to-noise, of the representation of motion in MT. We predict (and preliminary data confirm) that reduced dot coherence causes a reduction in the amplitude of the MT population response while keeping the preferred speed at the peak of the population response fixed at target speed (Stuart Behling & Stephen Lisberger, unpublished data). We predict that FEF_SEM produces lower output, and therefore a lower gain of visual-motor transmission, during pursuit initiation and steady state because the reliability of the motion signal is lower. We also predict (as yet untested) that the gain of the positive feedback of eye velocity through the cerebellar floccular complex is reduced by lower dot coherences.

In the early part of this century, our approach to understanding and modeling the pursuit circuit was transformed by our realization of the explanatory power of trial-by-trial analysis of behavior and neural responses. This occurred after three decades of thinking in terms of “lumped” models that reproduced averaged behavioral data (e.g., as in Figures 1B and 3C), and it changed how we do our experiments as well as how we think.

Models designed to reproduce behavior data have limited power of explanation for how neural circuits generate sensory-motor behavior. To understand the origins of variation in pursuit initiation, we therefore turned to analysis of variation in neural responses. Our first finding, which does not yet appear in a neural circuit model, was a surprisingly large trial-by-trial correlation (neuron-behavior correlation) between the firing rate of Purkinje cells in the floccular complex of the cerebellum and the eye speed in the initiation of pursuit (Medina & Lisberger, 2007). In some Purkinje cells, the correlation coefficient was as high as 0.9, meaning that we could predict almost all of the variation in eye speed from the best correlated Purkinje cells. The average correlation was greater than 0.6.

It seems inescapable that the actual influence of each individual Purkinje cell on motor output is much smaller than suggested by the neuron-behavior correlations. Floccular Purkinje cells have disynaptic connections to motoneurons, but we estimate that more than 1000 Purkinje cells participate in each of horizontal and vertical pursuit. Instead, we attribute the remarkably high neuron-behavior correlations to coordination across the population of Purkinje cells. In the few pairs we were able to record, we confirmed neuron-neuron correlations between Purkinje cells that could account for the neuron-behavior correlations. This led to the understanding that high neuron-behavior correlations do not reveal the strength of influence of a given neuron on the behavior. Rather, they indicate correlation within a population of neurons that are collectively driving the behavior. Indeed, a neuron could show a large neuron-behavior correlation entirely through correlations with other neurons, without having any physical connection, itself, to the motor pathways for the behavior (Chaisanguanthum, Joshua, Medina, Bialek, & Lisberger, 2014).

Correlated variation in MT seems to be an important cause of trial-by-trial variation in the initiation of pursuit. The evidence for this statement is that neurons in MT have neuron-neuron noise correlations (Bair, Zohary, & Newsome, 2001; Huang & Lisberger, 2009) and neuron-behavior correlations (Hohl, Chaisanguanthum, & Lisberger, 2013) during the initiation of pursuit (“MT-pursuit correlations”). In Figure 4B, for example, the symbols show data from multiple single trials for recordings from a pair of MT neurons. The graph plots the z-scored firing rate of one neuron versus another, revealing one example of impressive neuron-neuron noise correlations. Across the population of MT neurons, statistically significant neuron-neuron correlations occur in many pairs (see Figure 4E, red symbols) and correlations are larger for pairs of neurons with similar preferred speeds. The existence of neuron-neuron noise correlations does not alone mean that correlated sensory noise is a major cause of behavioral variation. The finding of MT-pursuit correlations does. We revealed MT-pursuit correlations by recording spikes in MT neurons and the simultaneous initiation of pursuit for up to 500 repetitions of the same visual stimulus (see Figure 4F). Many MT neurons had statistically significant MT-pursuit correlations that ranged as high as 0.4 and averaged 0.2 (see Figure 4G, open circles).

Computational modeling showed that the neuron-neuron correlations in MT could create both the MT-pursuit correlations and the variation in pursuit (Huang & Lisberger, 2009; Hohl et al., 2013). Further, the presence of large MT-pursuit correlations argues strongly that much of the behavioral variation arises from the sensory representation. Noise is possible downstream, of course, but it has conflicting effects on MT-pursuit correlations and behavioral variation (Schoppik, Nagel, & Lisberger, 2008; Egger & Lisberger, 2022). If noise is added downstream, then behavioral variance will increase relative to what would be caused by a given amplitude and structure of neuron-neuron correlations in MT (see Figure 4H, black line). At the same time, noise added downstream will create behavioral variance that cannot be attributed to correlated noise in MT and would decrease MT-pursuit correlations (see Figure 4H, red line). As we will see, there could be an exact balance of correlated noise in MT and downstream noise that will reproduce our data on both behavioral variation and MT-pursuit correlations.

We made the next step toward a more biologically motivated model of the sensory decoder for pursuit based on a simple observation about the relationship between the variance and mean of eye velocity in the initiation of pursuit (Egger & Lisberger, 2022). Increasing the size of a tracking stimulus from 2 to 6 to 20 degree breaks the traditional relationship (Harris & Wolpert, 1998) between the variance and mean of motor output (see Figure 5D). “Signal dependent noise” no longer holds, meaning at least that the source of noise cannot be in the final motor pathways and instead must be in the sensory representations and the sensory decoder for motor control.

We reproduced a large body of data with a model (Egger & Lisberger, 2022) where the sensory decoder explicitly includes two pathways (see Figure 5A). In the decoder, noise arises from both a correlated variation in the MT population response, as before, and in the gain control pathway. With reasonable assumptions, the model was able to reproduce accurately the magnitude of variation in the initiation of pursuit, the effects of increasing the size of the pursuit stimulus on the mean and variance of eye velocity at the initiation of pursuit, and the observed trial-by-trial variation in MT responses. We included model MT neurons with a wide range of tuning properties (see Figure 5B), appropriate neuron-neuron correlations (see Figure 5C), and realistic effects of increasing the size of the moving stimulus (Born & Tootell, 1992; Pack, Hunter, & Born, 2005). The equation for estimating motion reliability comprised a vector sum of the MT population response with each neuron's response weighted by the log of its preferred speed. The equation for estimating target speed comprised a vector average that depended somewhat on the amplitude of the population response, as in equation 3.1. The model does an excellent job of reproducing the mean eye velocity at the initiation of pursuit as a function of target size and speed (see Figure 5G).

If the gain control pathway is endowed with noise that is independent of the correlated variation in MT responses, then the model reproduces both the effect of target size on the variance of eye speed (see Figure 5F) and the amplitude and distribution of MT-pursuit correlations (see Figure 5I). If the gain control is noiseless, the model predicts a variance of eye speed that is unrealistically low and does not depend on target size (see Figure 5E). The model with noiseless gain control also predicts MT-pursuit correlations (see Figure 5H) that are considerably larger than those recorded in the monkey (see Figure 4G). Thus, the two-component sensory decoder has considerable predictive value. In addition, it makes the testable prediction of strong trial-by-trial correlations between the activity of FEF_SEM neurons and the eye speed at pursuit initiation (Schoppik et al., 2008), but without correlations between the activity of neurons in FEF_SEM and MT.

A larger effect of context for sensory data of lower reliability is exactly what is expected of a system the shows Bayesian-like behavior. Further, computational modeling shows that it is possible to understand the implementation of Bayesian-like behavior in terms of control of sensory-motor gain (Darlington et al., 2017). Expectations of a faster target speed should increase the firing of neurons in FEF_SEM and lead to a higher gain of visual-motor transmission. Lower-contrast targets should cause lower firing in FEF_SEM and lead to a lower gain of visual-motor transmission.

Recordings from FEF_SEM reveal all the components of the reliability-weighted combination of priors and sensory data and support the idea that gain control plays an important role in the associated pursuit behavior (Darlington, Beck, & Lisberger, 2018). During recordings, we placed the monkey alternately in 50-trial blocks of fast- or slow-context trials. In each trial, the monkey fixated for 800 to 1600 ms before he had to pursue a moving target. During the fixation interval, we saw a neural correlate of a prior based in expectations. Firing rate ramped up in preparation for the onset of target motion and reached a level that was higher in the fast context compared to the slow context. This finding appears in traces in Figure 6D that are aligned so that they end at the termination of fixation and the start of target motion. During the initiation of pursuit (see Figure 6E), neurons in FEF_SEM show evidence of the same combination of priors and sensory data as does pursuit behavior. Here, the traces start at the end of fixation and the start of target motion. For high-contrast targets, the response is essentially the same in both the fast and slow contexts (green versus blue continuous traces). For low-contrast targets, the response to low-contrast targets is both delayed and reduced in amplitude relative to that for high-contrast targets. Thus, the output of FEF_SEM during pursuit initiation is appropriate to control the gain of visual-motor transmission in a way that implements Bayesian behavior. The gain signal depends weakly on the prior when the sensory data come from high-contrast targets and are highly reliable; it depends more strongly on the prior when the sensory data come from low-contrast targets and are less reliable.

A simple neural network with fully recurrent connections among excitatory and inhibitory model neurons (see Figure 6C) reproduces all the features of our recordings from FEF_SEM. The network receives two inputs: a pulse of activity from visual motion processing with an amplitude that depends on stimulus contrast and a step of activity to indicate the time of fixation. The average response of the model units (see Figure 6G) shows preparatory activity that increases steadily during fixation. When the context changes from control to slow to fast and back to control, the preparatory activity in the model follows the same trajectory as the data (Figure 6H) because of plasticity placed strategically at the recurrent synapses from excitatory neurons. Thus, preparatory activity learns based on experience and reaches a peak at the time of motion onset (see Figure 6G, vertical arrow) that is larger in the fast versus slow context. After motion onset, the model neurons show a pulse of firing that is larger for high- versus low-contrast targets (continuous versus dashed traces) and reaches a higher peak firing in the fast- versus slow-context (red versus black traces). As in the data (see Figure 6K), the size of the pursuit-related pulse of activity in the model (see Figure 6L) is related to and larger than the modulation associated with the preparatory activity.

One theme of our research in the past few years is the value of trial-by-trial analyses in constraining and guiding efforts to create a computational model of pursuit. In this regard, it is important that the network model of FEF_SEM uses shunting inhibition to create the same trial-by-trial correlations between preparatory and pursuit-related activity as do the data (Darlington et al., 2018). Our analyses defined three different measures of the pursuit-related pulses of firing rate (see Figure 6F): the absolute modulation of firing rate (“absolute”), the preparatory modulation of firing rate (“prep”), and the incremental modulation from preparatory to the peak (“incremental”). Analysis of single-trial firing rates for FEF_SEM neurons showed that the absolute modulation of firing rate was correlated positively with the preparatory activity (see Figures 6I and M). In contrast, the incremental change in firing was correlated negatively with the preparatory activity (see Figure 6N). All model neurons in the successful network showed the same trial-by-trial effects (see Figure 6J). Thus, a generic cortical neural circuit model can, with a few strategic additions of plasticity and shunting inhibition, reproduce the activity of neurons that represent all of the components of Bayesian-like behavior.

Importantly, we do not think of the network in Figure 6C as a “Bayesian” circuit, nor do we think that it is particularly significant that the pursuit system can be seen as having Bayesian-like behavior. Rather, we see the terminology of “Bayesian” as a metaphor for a reliability-weighted combination of sensory data with adaptable expectations based on experience. We conclude that the existence of a gain-control mechanism in pursuit allows the FEF_SEM to play a key role in (1) regulating the pursuit response under different conditions of expectations and movement preparation, (2) assessing the reliability of the motion signals for pursuit, and (3) allowing the system to combine sensory data with expectations in an optimal manner.

Over the past 40 years, our approach to modeling the pursuit system has evolved to become more oriented toward biologically motivated or biomimetic models. We have moved from control theory models that aimed only to account for average pursuit behavior, to models that are based on biologically motivated model MT population responses, to models that recognize the complicated, parallel, multicomponent organization of the sensory-motor decoder. We have moved from models that reproduce average responses, to models that reproduce single trial responses, to models that are constrained by trial-by-trial neuron-neuron and neuron-behavior correlations. This is progress, but I would say that we have taken baby steps rather than large, bold strides.

To conclude this review, I evaluate the requirements for a biomimetic model, trying to be forthright about where we are, to outline where we need to be, and to offer a strategy for getting there.

2. The model needs to reproduce neural responses and behavioral data across a wide range of behavioral paradigms. Questions that need to be answered include at least the direction and speed tuning properties at different nodes of the circuit, the impact of stimulus form (especially motion reliability), and the presence or absence of preparatory activity (Darlington et al., 2018; Darlington & Lisberger, 2020, 2022). It also would be helpful to use brief pulses of target motion under different conditions to assess whether and how a given neuron participates in the neural mechanisms that control the gain of visual-motor transmission. Does each specific neuron sit before gain control, after gain control, or as part of gain control? Again, we have published sufficient data from MT and FEF_SEM, and we have preliminary data from NRTP and DLPN. But we do not know whether Purkinje cells in the floccular complex show the preparatory activity we see elsewhere in the system, or whether their responses scale as expected with eye velocity under different conditions of visual-motor gain. Modern recording technology with multicontact probes affords better stability than previous approaches with single electrodes, as well as faster accumulation of neural populations. We need to return to areas such as the floccular complex, the basal ganglia, and the oculomotor vermis to understand their responses across a wide range of behavioral and stimulus conditions, preferably with some degree of cell-type classification.

3. The model needs to include local circuits that perform local computations, as well as recurrent connections that deliver corollary discharge signals from the motor pathways back to the cerebral cortex and basal ganglia. In the past, we have focused on the macro-organization of the feedforward pursuit circuit with a skeleton diagram like Figure 7. One of the lessons from our neural network model of FEF_SEM (Darlington et al., 2018) is that important computations are performed by microcircuits within nodes of the circuit. Perhaps the diversity of response profiles within each node provides a clue to the computations performed within each node; perhaps identifying unique response properties in projection neurons will reveal which neurons are local interneurons and unlock the analysis of those local computations. A second lesson from FEF_SEM is the need to consider recurrent feedback of corollary discharge from the motor pathways. Many neurons in FEF_SEM discharge a pulse of activity during pursuit initiation and show sustained activity during steady-state tracking (Tanaka & Lisberger, 2002). Where does the sustained activity come from? Is it generated autonomously in FEF_SEM, or does it originate in recurrent feedback from the motor system such as we have demonstrated in the floccular complex of the cerebellum? The same question exists for other nodes, including the oculomotor vermis, basal ganglia, NRTP, and DLPN.

4. The model needs to operate in negative feedback configuration where the act of tracking alters the image motion input to the system and largely eliminates image motion during steady-state tracking. The model needs to account for eye velocity and neural firing rate as a function of time. Ours and others’ initial control theory models included time and dynamics and operated in feedback configuration, but this got lost as we attempted to transition to models that had more biologically realistic internals. The early models identified a number of features that are challenging to explain—for example, absence of overshoot at the initiation of pursuit (Krauzlis & Lisberger, 1989); spontaneous oscillations at higher frequencies than predicted purely by image velocity inputs (Goldreich et al., 1992); and essentially perfect steady-state tracking, eliminating the image motion drive to the system (Morris & Lisberger, 1987). Our first attempt to replace the control theory representation of visual motion signals with a realistic MT population response that varied as a function of time was only partially successful (Lisberger & Movshon, 1999). Instead, our recent, more “biologically realistic” models have represented neural responses as static spike counts rather than as time-varying firing rates or spike trains. We need to return to models with feedback and temporal dynamics.

5. The model needs to have realistic trial-by-trial variation, neuron-neuron correlations, and neuron-behavior correlations. We started 40 years ago with the idea that we needed to reproduce average data. More recent efforts (e.g., Figure 5) show the value of the single-trial data and simultaneous recordings from multiple neurons as constraints on models. Especially powerful are constraints that play off against each other, as do the variance of pursuit initiation and the amplitude of MT-pursuit correlations. I think that trial-by-trial data from other areas, including analysis as a function of time, hold promise for constraining future, more biologically realistic models.

Given the need for new data from more nodes of the pursuit circuit using a broader set of behavioral paradigms and stimulus speeds and forms, it would be reasonable to ask whether data collection should precede further development of a biomimetic computational model of the pursuit circuit. I think not. I think that it would be constructive to create a computational model of pursuit now, even given fragmentary data.

Finally, I ask whether the behavioral system and animal model I have chosen for study holds the correct promise for achieving a biomimetic circuit model of a sensory-motor behavior. In the 1970s and 1980s, we believed that recordings from single neurons in awake, behaving monkeys held the key to understanding how the brain works. In the 1990s, in the spirit of Neural Computation, we added computational modeling to our tool kit. Now we have optogenetics, calcium imaging, and multicontact recording probes that are deployed most easily and very commonly in mice.

I see a gap here. It is a challenge to apply modern technologies in nonhuman primates, and at least for some years, there is going to be a shortfall in the cell specificity of recording and stimulation. At the same time, I think there is great virtue in the strategy of adopting a highly quantifiable and controllable behavior and studying in quantitative detail the first- and second-order statistics of the relationships between neural responses and behavior. I do not see this happening in mice, at least not yet with the level of behavioral precision and granularity that is possible in the eye movement or arm movement system of nonhuman primates. In due course, NeuroPixel probes will be a game changer for analysis of circuit function in primates. They will allow simultaneous recordings of many neurons on potentially dozens of probes across multiple nodes of a single circuit. The resulting careful observations of the neural engine in action, in conjunction with quantitative analysis of well-understood behaviors, is arguably our most potent asset when trying to understand interesting neural computations. The interpretational power of neuron recordings in the eye movement system of monkeys is huge because we know how the discharge of neurons in many sites in the brain affects the activity of motoneurons to specific muscles. In the long run, I see a combination of modern technologies, quantitative analysis of behavior, recordings from multiple single neurons simultaneously in nonhuman primates, along with computational modeling with biomimetic neural circuit models, as an essential, critical component of neuroscience research.

I am grateful to many collaborators, graduate students, postdoctoral fellows, and technical staff for their contribution to my research program. I also thank the funding agencies that have supported the research, including the National Institutes of Health, the McKnight Foundation, and the Sloan Foundation. Current support is from NIH grant R01-EY027373.