Compensating for sensorimotor noise and for temporal delays has been identified as a major function of the nervous system. Although these aspects have often been described separately in the frameworks of optimal cue combination or motor prediction during movement planning, control-theoretic models suggest that these two operations are performed simultaneously, and mounting evidence supports that motor commands are based on sensory predictions rather than sensory states. In this letter, we study the benefit of state estimation for predictive sensorimotor control. More precisely, we combine explicit compensation for sensorimotor delays and optimal estimation derived in the context of Kalman filtering. We show, based on simulations of human-inspired eye and arm movements, that filtering sensory predictions improves the stability margin of the system against prediction errors due to low-dimensional predictions or to errors in the delay estimate. These simulations also highlight that prediction errors qualitatively account for a broad variety of movement disorders typically associated with cerebellar dysfunctions. We suggest that adaptive filtering in cerebellum, instead of often-assumed feedforward predictions, may achieve simple compensation for sensorimotor delays and support stable closed-loop control of movements.
The apparent ease with which we perform most daily tasks masks the difficulties of movement control. Even simple tasks such as reaching to grasp an object or performing a saccade toward a visual stimulus are supported by efficient closed-loop neural control. Several challenges arise for the brain, of which the following two have been the focus of much research effort: the compensation for sensorimotor delays and the filtering of sensorimotor noise. Although these two functions have often been dissociated experimentally, theoretical considerations and recent results indicate that estimation and prediction are performed simultaneously in the brain.
On the one hand, sensorimotor delays are too long to explain skillful motor behavior based on delayed-feedback control only (Crevecoeur & Scott, 2014; Kawato, 1999), which suggests that the brain uses internal models and predictions (Miall & Wolpert, 1996; Wolpert & Ghahramani, 2000). One proposed mechanism for predictive control was the Smith predictor (Miall, Weir, Wolpert, & Stein, 1993). In this framework, a model of the biomechanical system is used to predict the consequences of motor commands ahead of sensory feedback. Movements are then controlled based on internal estimates, which removes the delay from the feedback loop provided that the internal model is correct (Zhong, 2010). One difficulty with this approach, and with other prediction techniques, is that they are excessively sensitive to model errors, such that an arbitrarily small mismatch in model parameters can lead to unstable closed-loop control (Michiels & Niculescu, 2007; Mondié & Michiels, 2003). Clearly, assuming that the brain has perfect knowledge of the peripheral motor system is not realistic. Thus, although the Smith predictor captures the principles of predictive neural control, it is unlikely to be implemented in neural circuits as such. Another compensatory mechanism must be at play to mitigate the impact of prediction errors.
On the other hand, with regard to sensorimotor noise, a wealth of research has shown that the brain uses near-optimal estimates of variables such as hand location or of forces applied to the limb, obtained by combining prior knowledge with available sensory information in a statistically efficient manner (Angelaki, Gu, & DeAngelis, 2009; Koerding, 2007). This function has been described in the framework of Bayesian statistics in numerous studies on perceptual or motor decisions (Wolpert & Landy, 2012), but also, and importantly, during online control (Izawa & Shadmehr, 2008; Kording & Wolpert, 2004; Wolpert, Ghahramani, & Jordan, 1995). While these previous studies highlighted an efficient handling of uncertainty in the environment and in the nervous system, it is still critical to take sensorimotor delays into account to avoid unstable closed-loop control (Crevecoeur & Scott, 2013). Consistent with this hypothesis, recent results have shown that the combination of internal priors with sensory feedback depends not only on the variance of the feedback signals but also on their temporal delays (Crevecoeur, Munoz, & Scott, 2016).
Thus, predictive control requires compensation for prediction errors, and optimal estimation must include a compensation for temporal delays. In theory, an optimal (Bayesian) integration of prior and delayed feedback is achieved by estimating the present and past states, which for discrete time systems can be achieved in an augmented system (Challa, Evans, & Wang, 2003). The integration of multiple, delayed measurements can also be optimally integrated using the same approach. However, system augmentation is suitable for digital control, and it is difficult to translate it to continuous time systems or to systems for which present and past estimates may not be jointly available, as in the sensorimotor system. In such a case, when optimal delay compensation cannot be achieved, prediction errors unavoidably occur, and the question arises as to whether closed-loop control remains feasible. Here we investigate this question in theory by analyzing the impact of approximate delay compensation in closed-loop control models of eye and arm movements subject to multiplicative noise in the command (Jones, Hamilton, & Wolpert, 2002) and affected by temporal delays compatible with each motor system.
A seminal study by Miall and colleagues (1993) addressed how a mismatch between the controller and plant delays affected the behavior of simple systems, such as low-pass filters and integrators combined with single gain control. This study also considered relatively long delays (150 ms) in comparison with visual (100 ms) or somatosensory delays (50 ms in the upper limb) (Scott, 2016). In addition, this formalism did not include stochastic disturbances or uncertainty in the process, which is known to affect planning and control (Harris & Wolpert, 1998; Todorov & Jordan, 2002). To address these limitations, we revisit the problem of predictive control subject to prediction errors in stochastic control models of eye and arm movements. Our approach to this problem is based on finite spectrum assignment (FSA) (Michiels & Niculescu, 2007; Niculescu & Gu, 2004), which in the context of linear and deterministic systems can be stabilized by using a low-pass filter (Mondié & Michiels, 2003). Following the approach of Mondié and Michiels (2003), we show that probabilistic filtering of the predicted state can compensate for prediction errors, including errors that result from a low-dimensional approximation of the system dynamics or from a delay mismatch between the observer and the true system.
Our analyses of the interplay between filtering and prediction errors provide insights into biologically plausible control mechanisms. Indeed, we show that delay compensation may be performed by low-pass filtering of motor commands over the delay interval (such as a moving average), which is easy to link to adaptive filtering in neural circuits. Furthermore, we illustrate how combining priors and predictions improves the stability margin of predictive control and how errors in the delay generate over- or undershoots qualitatively comparable to human movements. Thus, approximate delay compensation can still generate efficient movements in human-inspired models. In addition, this approach captures explicitly the influence of both uncertainty and time in state estimation, which represents a useful framework for modeling the impact of these variables during sensorimotor control. Finally, we found that prediction errors generated a wide spectrum of movement dysmetria that typically characterize disorders observed in disease and, in particular, in cerebellar ataxia and essential tremor. In this regard, adaptive filtering for stable closed-loop control may describe the function of cerebellum more accurately than the often-assumed feedforward predictions.
We first present the problem definition and provide the derivation of the control algorithm. The section continues with an illustration based on a Langevin process, which shows analytically how the model captures the dependency of the optimal filter on both signal variances and temporal delays. We then illustrate the performance of the algorithm on bio-inspired models of eye and upper limb movements to investigate the robustness of this control algorithm against prediction errors.
2.1 Definitions and Assumptions
The variable represents an -dimensional zero-mean gaussian random variable with covariance . Thus equation 2.2 expresses that the current sensory feedback and the delayed state ( and , respectively) have joint gaussian distribution. Finally, we assume without loss of generality that is positive definite.
The problem consists in deriving a feedback controller that minimizes a quadratic cost function. This problem can be solved using linear quadratic gaussian control (LQG) (Astrom, 1970; Phillis, 1985; Todorov, 2005), which gives a control law of the form , where is an estimate of the present state of the system. The challenge is to derive the optimal estimate in the presence of noise and measurement delays.
2.2 Optimal Solution Based on System Augmentation
This approach allows optimal compensation for temporal delays; however, it is clearly an artificial technique that is suitable for digital applications. First, it is based on the key assumption that the delay is fixed and known, which is not desirable in this context, as we are interested in how the nervous system can handle errors in the prediction or in the estimate of the delay. In addition, the delay compensation is performed by updating the time series included in through the block structure of the state estimator, which requires storing present and past estimates. In this study, we investigate the impact of prediction error that may arise from approximate representation of the dynamics during the delay interval or from errors in the estimate of the delay. Thus, we derive a near-optimal closed-loop controller, that allows manipulating the delay compensation directly. It is built on the insight that system augmentation by design achieves extrapolation of delayed sensory information to the present. Our proposed approach is thus based on an explicit extrapolation of the state measurements.
2.3 Filtering over Predictions
Here we derive an estimator that extrapolates the current sensory feedback explicitly, prior to performing optimal state estimation. Our approach is based on the premise that for deterministic systems, low-pass filtering of the predicted state is known to stabilize predictive control against prediction errors (Mondié & Michiels, 2003; Niculescu & Gu, 2004). We thus derive a similar estimate in the context of stochastic processes and use the statistics of the predicted state to design an optimal filter (Kalman filter) and mitigate the impact of prediction errors.
Note that the cross products of and are zero on average, and the expected value of is by definition equal to . The variance of has two terms. The first term captures the uncertainty associated with the measurement used as an initial condition. Indeed, it expresses how the measurement noise propagates through the systems' unforced dynamics over . It is worth mentioning that the uncertainty associated with decays exponentially, remains constant, or increases in the dimensions in which the matrix has negative, zero, or positive eigenvalues, respectively. The second term is the variance induced by the process noise, and thus it is a function of the system dynamics captured in , and dependent on .
The optimal filter over the predicted state is thus obtained by applying this result to calculate the posterior distribution of the next estimate given the current estimate and the extrapolated measurement (see equations 2.6 and 2.7). The whole estimation algorithm is as follows:
- The initial state estimate is assumed to have known gaussian distribution:
- The true state trajectory over a small discretization interval of iswhere , , and . With this definition, we have that .
- The one-step prediction of the state at the next time step is computed by simulating the system dynamics over one time interval of :
Finally, we use the expression of the conditional distribution of gaussian random variables given above to combine the one-step prediction ( with the extrapolated state (, which gives the following recurrence and completes the definition of the estimation algorithm:
Observe that the filter is well defined because is positive definite from assumptions and is nonnegative; hence, is also positive definite. This can be verified by expanding equation 2.13 and using the identity that in the sense of Loewner ordering. Thus the matrix inverted in equation 2.11 is positive definite. Observe also that this filter is equivalent to a standard Kalman filter when there is no delay, as equation 2.5 computed when gives . It is thus a natural extension of the Kalman filter to systems subject to measurement delays.
The filter defined above is based on a prediction of the state for simplicity (FSA), as we consider a state-feedback controller. The difference between FSA and the Smith predictor is that FSA predicts the state of the system, whereas the Smith predictor predicts the output signal, which can be a linear function of the state. With the output signal of equation 2.2, these two approaches are identical in the context of this letter. When a more general output equation of the form is considered, it is necessary to reconstruct the state vector at with an observer prior to the extrapolation step (see equation 2.5). Note that such an observer need not be optimal to derive the filter above, provided that its output corresponds to equation 2.2 as is the case for general observers.
2.4 Optimal Control
Thus far, we have derived an optimal filter for the predicted state, assuming that the control function , was known. We now derive the optimal controller separately assuming that an optimal estimate of the state is available through the recurrence above (see equations 2.10 to 2.13). It is important to emphasize that this approach is based on a heuristic assumption that the separation principle applied, allowing us to solve the estimation and control problems independently. However, this assumption is not true in general. For instance, when considering multiplicative noise as in sensorimotor systems (e.g., , the optimal control and optimal filter depend on each other. In the linear case, the estimator and controller can numerically be approximated by iterating the sequence of nonadaptive Kalman gains and control gains until convergence (Phillis, 1985; Todorov, 2005). Our approach is based on first deriving the control gains as if the separation principle applied, and then optimizing the Kalman gains relative to the ongoing control function in each simulation run through the time-varying . Thus, our approach is globally suboptimal for linear systems, but we show in the next section that the loss is not substantial. The advantages are that it allows adapting the estimate online following modeled but unexpected disturbances without recalculating the control and estimation gains. It also handles general noise functions captured in .
2.5 Low-Dimensional Approximation and Implementation
Thus far, our developments consisted in replacing the augmentation technique by an explicit extrapolation of the state measurement. The dimensionality of the augmentation corresponds to the number of time steps in the quadrature rule presented in equation 2.16 when the same discretization step is used. Importantly, this dimension can be reduced by approximating the sum in equation 2.16 in one step, which can be done in theory, and approximated in practice with a summary version of , for s. These expressions will allow us to investigate the robustness of the closed-loop controller against extrapolation errors. In addition, we will also investigate the robustness against an error in the internal estimate of the delay by assuming that the controller uses , which has an impact on the estimate in equation 2.17 through the matrices and .
We will show that the state estimator defined in equations 2.10 to 2.13, combined with the low-dimensional quadrature rule given in equation 2.17, constitutes a good candidate model of neural compensation for sensorimotor delays. Indeed, this algorithm involves only multiplications of matrices and vectors, along with an operation that approximates over . A schematic block representation of this estimator is shown in Figure 1. The inputs to this pathway consist of the motor command, available through the corollary discharge, the state estimate, and the sensory feedback, which are all available in the brain (Shadmehr, Smith, & Krakauer, 2010; Wolpert, Diedrichsen, & Flanagan, 2011; Wurtz, 2008). The operations are matrix-vector multiplications and additions, plus the averaging, or low-pass filtering, represented by the rectangular box. The remaining computational operations consist in adjusting over time as a function of according to equations 2.11 and 2.13. In order to validate this schematic pathway for studying the impact of prediction errors during motor control, we must show that it generates behaviors compatible with biological movements. This is done in the next section.
3.1 Langevin Process
It can be observed that is equal to when , and it is asymptotically equal to as . Thus, increases, remains constant, or decreases with the delay, dependent on whether is smaller than, equal to, or greater than . The fact that the extrapolation variance decreases over time when is counterintuitive, as it suggests that the extrapolation becomes more reliable with longer delays. This is true in principle; however, the reason the extrapolation becomes more reliable is simply that it tends to zero due to the stable dynamics. As a consequence, the true state and the estimate converge to zero. In any case, even when , it is always useful to integrate measurements of the state in the state estimator because and are correlated (see equation 2.6), and thus the projection of onto the linear space generated by will always decrease the estimation error.
3.2 Simulation of Human Movements
We now present simulations of saccadic eye movements and single joint reaching of the upper limb with mechanical perturbations. These examples were chosen because the biomechanical plant can be approximated in a linear model and thus correspond to the class of stochastic processes defined in equation 2.1. The extrapolation was based on the two rules given in equations 2.16 and 2.17. We then varied the delay parameters used in the estimation (, without varying the true delay, in order to investigate numerically the sensitivity of these control models to this parameter.
Eye movements are of particular interest as the visual system is affected by relatively long delays and the eyeball has low inertia. Thus, close-loop control of the oculomotor plant is challenging and likely more sensitive to model errors than other motor systems. The simulations shown below use ms for the oculomotor system and ms for the upper limb. For the visual system, this delay is based on the latency of reflexive saccades (Munoz & Everling, 2004). For the upper limb, the delay of 50 ms corresponds to long-latency feedback supported by a pathway through cortex (Scott, 2016). Details about the models and parameters can be found in the section 5.
We first evaluate the performance of the algorithm derived above by comparing the results with those obtained in the extended LQG framework with system augmentation (Todorov, 2005) in order to quantify how it departs from the augmentation technique (see equation 2.3). When the prediction is performed with the high-dimensional quadrature rule, which corresponds to the dimension of an estimator obtained with state augmentation (see equation 2.16), the average behaviors of the two control algorithms are almost indistinguishable (see Figure 2a, black and blue traces). The difference can be observed in the peak positional variance, where an increase of about 10% can be noticed. The positional variance then recovered similar values as in the case of extended LQG near the end of the simulation (see Figure 2b). We also measured the cost for each trajectory and found an average increase of less than 10% in comparison with the measured and predicted expected costs from extended LQG. This result is expected because our controller uses slightly greater control signals, which in turn induces more variability across simulations due to signal-dependent noise.
Comparing the two techniques, the augmentation reduces the system to the nondelayed case, but in this context, the presence of signal-dependent noise requires approximate solutions. In contrast to the explicit delay, compensation provides an exact filtering solution to a modified problem and for a given control law. In particular, the analyses in Figure 2 show that for behaviors considered in this study, the simulations obtained for both controllers are very close, and thus the filter developed in equations 2.10 to 2.13 is valid and suitable for studying approximate delay compensation in biological control.
The impact of the low-dimensional quadrature rule is also shown in Figure 2. The main effect is a reduction in the control gain, followed by larger target overshoot for the eye trajectory (see Figure 2a, red). The consequence is that the traces become less variable across trials due to reduced signal-dependent noise, whereas the average cost displays approximately a three-fold increase. We verified that the results were similar for the reaching task (simulations not shown). Note that our goal is not to establish a novel control model for which solutions are available (Bar-Shalom, Huimin, & Mahendra, 2004; Challa et al., 2003; Choi, Choi, & Chung, 2012; Todorov, 2005) but to investigate the impact of extrapolation errors, as well as of errors in the estimate of the delay. To this aim, below we use the low-dimensional quadrature rule (see equation 2.17) and modify the delay parameter used by the controller to compute the extrapolation.
Figure 3a illustrates the impact of an error in the internal estimate of the delay. The different traces were averaged across 50 simulation runs with an estimate of the delay ( that was 30% or 50% of the true delay chosen for illustration (left and right, respectively). Simulations are shown with the closed-loop controller using sensory predictions only (gray, , and sensory predictions with filtering (black, from equation 2.12) to highlight how filtering reduces the impact of prediction errors. It is noticeable that varying this single parameter generates different behaviors dependent on whether the delay is over- or underestimated. In the case of underestimation (left), the true trajectory is ahead of the estimated trajectory, leading to overshoots and oscillations in the eye angle. In contrast, when the delay is overestimated (right), the controller generates trajectories that fall short of the target and gradually catch up over time. Qualitatively, the simulation of a pursuit movement displays similar properties, although the impact of errors is not as important as during saccades because the pursuit task uses lower control gains than saccadic movements.
To quantify the oscillatory behavior in the simulations, we extracted the ratio between the first and second peak velocity based on the fact that the eye plant was a second-order model (see Figure 3b). It can be observed that the filtering limits the amplitude of the oscillations, which reproduces the results previously established by Mondié and Michiels (2003) in the context of stochastic control. These authors showed that filtering mitigates the impact of prediction errors induced by a quadrature. Here we directly apply this result to stochastic processes and show that prediction errors induced by an approximated extrapolation can also be stabilized. In principle, it is clear that filtering can only improve the controller; thus, the important and novel observation is that closed-loop control can be stabilized for a relatively broad range of delay errors in human-inspired models of eye and arm movements.
We also observed that the main frequency of the oscillations was lower when the filter was applied, and the difference was consistent across over- or underestimation of the delay (average reduction oscillation frequency of 25%). It is also noticeable that it is possible to minimize the amplitude of the oscillations by using an approximation of the delay (see the black arrow, Figure 3b); thus, the delay estimate and the low-dimensional approximation can be tuned together to achieve efficient closed-loop control. Together, these results highlight that applying the filter on approximate predictions improves the stability margin of the closed-loop controller.
In theory, equation 3.2 captures analytically (in the simplest case) the interplay between the noise properties of the process and the delay for estimating its state. In the simulation presented, we concentrated on prediction errors induced by the delay while assuming that the noise parameters were fixed. Interestingly the noise properties have different effects on the simulations. To observe this, we scaled the motor and sensory noise matrices by a factor of 0.5 and 2 for the saccade task, with a delay error of . In this range, changes in motor noise affected the variability of trajectories and estimates without affecting the closed-loop dynamics substantially. In contrast, increasing the sensory noise reduced the gain of the prediction error in the Kalman filter, which tended to reduce the oscillations induced by the prediction error (simulations not shown). A theoretical account of the impact of errors in the noise covariance matrices on state estimation can be found elsewhere (Heffes, 1966; Nishimura, 1966).
Simulations of upper limb control are shown in Figure 4 and emphasize similar results. Because the delay is shorter and the limb has higher inertia, the impact of using a low-dimensional quadrature rule was hardly visible (see equation 2.16). Figure 4 illustrates the effect of filtering by comparing trajectories with or without filtering for 20 degree reaching movements with a perturbation pulse applied during movement (1 Nm applied for 100 ms; the onset time is represented with the black arrow). The simulations were generated with varying levels of delay error. When the delay is overestimated (, the extrapolation is larger than the true state, and the overestimation in turn reduces the control gain. This effect results in shallower movements with slow corrections that do not oscillate until large errors are considered (simulations not shown). Thus, the analysis in Figure 4 concentrated on delay underestimation, which has more impact on control.
In this situation, when the true delay is underestimated (, the system starts oscillating because the extrapolated state is behind the true state and the controller erroneously increases the control gain. An example is given in Figure 4a for a value of the delay estimate chosen such that the control based on the predicted state only is unstable, whereas the filtered prediction with the same delay error is still stable due to a larger stability margin. The amplitude of the oscillations with or without filtering for delay underestimation is shown in Figure 4b.
To investigate the model properties in more detail, we sought to characterize the oscillations across a range of loading conditions. This simulation was clearly motivated by the variety of tremor types across clinical conditions (e.g., physiologic, essential), and we were interested to see if the model could provide insight into the origin of any kind of tremor. We simulated a posture task in which the controller had to maintain the limb at the origin angle ( and applied a small pulse at the beginning of the simulation (see Figure 5a). We focused again on delay underestimation as it evoked clearer oscillations and had more impact on control, but in biological control, it is clear that both over- and underestimation may occur. These simulations were computed with a relative delay error of 20%, in three distinct conditions of limb inertia ( Kgm, and 20%). For a limb of 2.5 Kg such as the wrist and hand, a change of 20% corresponds to a loading of 500 g, as used in previous reports on tremor (Elble, 1986). Interestingly, we found that the peak in the power spectral density was relatively invariant (less than 1 Hz difference) across the three loading conditions (see Figure 5b). The peaks occurred also at the same frequency when we varied the delay error (see Figure 5c). This is due to the fact that the system remains a delayed system with a term corresponding to the true delay for any (including arbitrarily small) delay error (Michiels & Niculescu, 2007). Qualitative similarities with essential tremor are discussed below.
Importantly, we found that for all simulated tasks, the compensation for sensorimotor delays was necessary. Indeed, using corresponds to a situation in which the sensory feedback (see Figure 1; is directly compared to the one step prediction of the next state ( without any delay compensation (see the red box in Figure 1). In this case, all simulations displayed unstable oscillations that increased in amplitude over time and eventually diverged toward infinity. This was observed even for simulations of reaching movements, despite the fact that the upper limb is less prone to instability due to shorter delays and higher inertia. These results suggest that it is necessary to compensate for delays when combining priors and feedback. Alternative control models are thus constrained by the fact that combining priors and feedback without delay compensation may lead to unstable closed-loop control. Previous work shows that it is possible to use delayed state–feedback control to model human motor corrections (Crevecoeur & Scott, 2014). However, since there is evidence of continuous state feedback control and of the existence of internal priors supporting motor responses (Crevecoeur & Scott, 2013; Crevecoeur & Kurtzer, 2018), these control models are not suitable to describe the neural mechanisms underlying long-latency responses to mechanical perturbations.
Finally, we observed that the approach presented could provide a heuristic for combining asynchronous and delayed measurements of the process, such as when vision and proprioception must be combined (Crevecoeur et al., 2016). This problem can in principle be solved based on system augmentation (Challa et al., 2003), which allows updating the posterior distribution of the present and previous states when a novel but delayed measurement becomes available. Here we sought to test whether independent extrapolations of two measurements could be combined with the filter already defined. Such independent extrapolation is suboptimal in theory because it does not correct simultaneously the present and previous estimates. However, it is relevant in the context of sensorimotor control because it remains unclear when or how sensory information from distinct modalities is combined in the neural feedback controller (Oostwoud Wijdenes & Medendorp, 2017). Thus it is interesting to ask whether suboptimal delay compensation may approach the optimal solution.
Using the measurement signal from equation 3.4 captures the fact that the latency and gain of feedback responses to perturbations are similar with or without the most delayed measurement, whereas the positional variance across simulations is reduced in the case of combined measurements. Such behavior was observed when participants were instructed to track visually the motion of their hand while their arm was mechanically perturbed (Crevecoeur et al., 2016). This latter study reported similar response latency but reduced estimation variance, which cannot be captured if the different sensor signals are linearly combined without taking their respective delays into account. However, we observed that this independent extrapolation process was numerically fragile and, with a low quadrature rule and the parameters considered, occasionally yielded unstable trajectories. The conditions under which approximate predictions in the form of equation 2.17 can stabilize control with multiple measurements delays should provide informative constraints on the accuracy of neural representations associated with eye-hand coordination.
To summarize, we have shown that combining prediction with optimal filtering can capture closed-loop control of human-inspired models of eye and upper limb movements. In this context, we have highlighted that sensory prediction errors arising when the process integral is approximated over the delay interval, or when the delay is not known exactly, generated movement over- or undershoots that can be partially mitigated by the application of a filter that considers the noise properties of the extrapolation and the process dynamics. Thus, combining priors with predictions about movement outcome improves the stability margin of the closed-loop controller.
We have studied the interplay between sensory prediction and state estimation in human-inspired models of the eye and upper limb movements. The presence of sensorimotor delays motivated the hypothesis that the nervous system uses predictions (Miall et al., 1993), often expressed as feedforward control (Kawato, 1999). However, even with predicted feedback in the case of the Smith predictor or with predicted state in the case of FSA, the system remains a closed-loop system, and delays remain problematic when the prediction is not perfect. The delay compensation involves two filtering operations. The first operation is the weighting of one-step predictions with the extrapolated state measurement, which takes the form of a standard Kalman filter (see equation 2.12). The second filtering operation is the computation of the extrapolated state measurement , which in theory requires integrating the system dynamics over the delay interval (see equation 2.5), but can be approximated with a moving average of the control function as in equation 2.17, or other classes of filters. We have illustrated based on simulations that an approximate extrapolation can still generate movements compatible with human movements, and we have highlighted how the Kalman filter reduces the impact of prediction errors by improving the stability margin. Thus, in addition to reducing uncertainty for perception, optimal estimation also critically improves the stability margin of predictive neural control.
Our approach was mainly motivated by evidence that the nervous system extrapolates sensory signals to adjust motor commands to the expected instantaneous state of the body or environment. Evidence for internal predictions during visual tracking is widespread. Indeed, previous studies showed that both sensory and extraretinal information about the eye and target trajectories are used during online tracking (Bennett, de Xivry, Barnes, & Lefevre, 2007; Blohm, Missal, & Lefevre, 2005; de Brouwer, Yuksel, Blohm, Missal, & Lefevre, 2002), or when anticipating future events such as trajectories following collisions (Badler, Lefevre, & Missal, 2010). This hypothesis also accounts for the perceptual loss that occurs around the time of saccades, suggesting that sensory extrapolation is an important component of visual processing (Crevecoeur & Kording, 2017). Regarding upper limb motor control, the same hypothesis accounted for the use of priors in motor responses to perturbations of distinct profiles (Crevecoeur & Scott, 2013). In this reference, long-latency responses (approximately 50 ms in the upper limb) depended on the expected profile, which is compatible with an extrapolation of the sensed perturbation-related motion. Thus, we included the extrapolation of sensory information explicitly in the model to account for this function of the nervous system (see equation 2.5).
A clear limitation of our developments is that we focused on linear systems. Of course, the neuromusculoskeletal system is nonlinear. Mathematical challenges aside, the algorithm proposed above is in principle also applicable to nonlinear models of the sensorimotor system. An interesting question for future studies is to investigate whether or how much filtering tolerates prediction errors that arise when one ignores the nonlinearities of the biomechanical plant.
In spite of its simplicity, a compelling aspect of the model is that it captured a broad spectrum of motor disorders by varying a single parameter. Indeed, assuming that the delay compensation mechanisms are similar for eye and limb movements, then movement dysmetria, including hyper- and hypometria, nystagmus, oscillations, and limb tremor, can be explained by assuming that the estimated delay is not equal to the true delay. In practice, such error simply comes down to an inaccurate calibration of the averaging or filtering operation that extrapolates the sensory feedback (see Figure 1, red box).
Regarding the neural basis of the delay compensation, cerebellum has often been associated with motor predictions and state estimation (Bastian, 2011; Kawato, 1999; Miall, Christensen, Cain, & Stanley, 2007; Miall et al., 1993; Wolpert, Miall, & Kawato, 1998). What reminds one of cerebellum again in the context of this letter is that, qualitatively, the dysmetria obtained by varying the internal estimate of the delay has been associated with cerebellar ataxia in clinical populations or following inactivation through cooling experiments performed with nonhuman primates (NHPs) (Bodranghien et al., 2016; Diedrichsen & Bastian, 2014; Mackay & Murphy, 1979). Regarding the control of eye movements, saccadic dysmetria as well as nystagmus were reported in cerebellar ataxia (Gomez et al., 1997; Leech, Gresty, Hess, & Rudge, 1977; Zee, Yee, Cogan, Robinson, & Engel, 1976). This is consistent with the idea that a wrong estimate of the delay in cerebellum alters saccade amplitudes through its involvement in the online control of saccades (Kheradmand & Zee, 2011), which in turn generates movement errors and oscillations.
Turning to the control of the upper limb movements, oscillations were observed in cerebellar patients following reaching movements (Diener, Hore, Ivry, & Dichgans, 1993), similar to those reproduced in Figure 4. Qualitatively, similar oscillations were documented in NHPs following inactivation of cerebellum (Flament, Vilis, & Hore, 1984; Hore & Flament, 1988; Hore & Vilis, 1984). As well, a range of deficits including hyper- and hypometria characterized goal-directed reaching in cerebellar patients (Bhanpuri, Okamura, & Bastian, 2014), a deficit that could be reproduced here by varying the estimated delay.
Of course, other parameters may generate undershoots or oscillations in movements such as errors in the parameters of the internal models (e.g., eye or limb inertia, viscous forces), and it is difficult to derive predictions that are specific to the compensation of sensorimotor delays, and not to other sources of errors. We focused here on approximate delay compensation and thus highlighted that errors in this parameter can also potentially generate movement disorders.
To investigate further the nature of oscillations induced in this model, we simulated a posture task and extracted the main frequency following small perturbation pulses (see Figure 5). Interestingly, we found a main frequency of approximately 4 Hz, which was invariant across loading conditions and errors in the delay. This result bears a striking similarity to previously documented cases of essential tremor, associated with main frequencies in the range 4–8 Hz, and independent of loading (Elble, 2003, 2013). These observations pointed to a neurogenic source of essential tremor that involves a cortico-cerebellar loop (Muthuraman et al., 2018). Thus, poor delay compensation captures oscillations of the closed-loop system at a frequency determined by the fixed and true delay and relatively insensitive to loading or internal errors. In this view, it is not clear that any brain region could be unambiguously identified as the source of oscillations in essential tremor, because the error can be small, but the details of processing can make the closed-loop system oscillate.
This framework is also consistent with the role of cerebellum in coordinating multiple effectors (Bastian, Martin, Keating, & Thach, 1996; Bo, Block, Clark, & Bastian, 2008; Diedrichsen, Criscimagna-Hemminger, & Shadmehr, 2007), as uncoordinated movements can also arise from errors in the estimated delay. Assuming that cerebellum hosts a pathway similar to that of Figure 1, then all of these motor dysfunctions can be explained by inaccurate tuning of the sensory extrapolation, corresponding in principle to an error in the estimate of the delay or, equivalently, an estimate that is not adjusted to the true difference between the sensory feedback and the current state of the body.
Finally, a strength of the model is that the low-pass or averaging operation that is required to compensate for the delay may be directly linked to the adaptive filtering performed in cerebellar microcircuits (Dean, Porrill, Ekerot, & Jorntell, 2010). Thus adaptive filtering of sensory extrapolations may achieve delay compensation and support predictive motor control. The model also suggests that the time constants of the adaptive filters in cerebellum should be related to the temporal delay and to the mechanical properties of the associated motor systems (see equation 2.17). To conclude, we suggest that errors in the estimated delays, or a poor tuning of the filtering operation required in the extrapolation of sensory feedback, may constitute the underlying cause of the wide variety of movement disorders generally associated with cerebellar dysfunctions.
5.1 Continuous-Time Models
The two models were augmented with the target location. The state vector for the upper limb was further augmented with an external torque ( representing the perturbation applied to the limb. This external torque is assumed to follow step functions, that is, . This assumption is necessary to reproduce responses to perturbations following a step function, as it must be estimated to generate active compensation for the sustained load applied to the limb. It is also compatible with the behavioral observations that internal priors about the perturbation profile influence long-latency responses (Crevecoeur & Scott, 2013).
5.2 Delays and Noise Parameters
For the upper limb control system, we added a prediction noise that affects the one-step prediction , without affecting the process dynamics. This prediction noise is required by the presence of an external torque ( that is not controllable through the change in control . Since there is no process noise affecting this variable and because it is not controllable, it is necessary to consider that there is uncertainty about this variable through the internal estimate to allow the estimator to track possible changes in this variable due to external disturbances. This prediction noise was assumed to follow a gaussian distribution with zero mean and covariance matrix set to 0.001 times the identity matrix of the same dimension as the state vector.
These parameters were adjusted manually so that the simulations were numerically well conditioned. Indeed, numerical errors can propagate through the recursions (see equations 2.10– 2.13), leading to bad conditioning of the matrix being inverted even if it is mathematically well defined. These noise parameters have an impact on the variability of the trajectories. In principle, the stability margin may depend on the noise parameters, but the improvement obtained with filtering should not depend on these values. A thorough characterization of the relationship between the noise parameters and the stability radius relative to the estimated delay is an interesting question for prospective work.
5.3 Cost Functions
For the reaching task (see Figure 4), we considered a cost function similar to equation 5.4 with for 300 ms following the movement time, during which was set to 0. The movement times for the reaching movements represented in Figure 4 was set to 500 ms. The posture task (see Figure 5) was simulated with a target angle of zero for 800 ms, and no movement time. The external perturbation was generated during each simulation run as a force pulse. The pulse amplitude was 2 Nm for the reaching task and and 0.5N for the posture task. In each case, the force pulse was applied for 100 ms.
We thank Andrew Pruszynski for comments on an earlier version of the manuscript. F.C. is supported by the F.R.S.-FNRS (Belgium).