Body Mechanics, Optimality, and Sensory Feedback in the Human Control of Complex Objects

Humans show remarkable dexterity in a wide range of skills, from juggling balls to a host of seemingly mundane actions in everyday life. For example, reaching for a glass of wine, swirling it, and leading it to one's mouth to drink involves interacting with—and controlling—the complex fluid dynamics acting on the hand. Handling such complex and potentially even chaotic dynamics is a feat that robots have failed to accomplish so far. Also in human motor neuroscience, most research on human movement control to date has only addressed relatively simple movements, such as moving one's hand from point to point, void of any dynamics arising from objects and interactions with the environment. Consequently, models of human motor control may have remained relatively simple as they did not have to deal with the complex and sometimes unpredictable interaction dynamics. Human manipulation of objects—the essence of tool use—has remained largely beyond neuroscientists' reach. To better understand how the brain controls and coordinates movements, we must look beyond unconstrained reaching movements and study richer and more naturalistic behaviors to reveal the underlying control principles.

Characterization of a dynamical system and its control requires exciting inputs that perturb the dynamical modes (Åström & Torsten, 1965; Åström & Eykhoff, 1971). Hence, the application of external perturbations has been the widely used method to interrogate the human motor system (Shadmehr & Mussa-Ivaldi, 1994; Krakauer et al., 2000; Burdet et al., 2001). Beyond controlled external perturbations, another way to excite the motor system and reveal its dynamical modes is to expose the motor system to interactions with the world as they are naturally inherent in daily activities. Only few previous studies examined more naturalistic tasks, such as playing pool billiards (Haar et al., 2020; Haar & Faisal, 2020), bouncing a ball on a paddle (Sternad et al., 2001), and interacting with dynamically complex objects, such as carrying a cup of coffee (Nasseroleslami et al., 2014; Maurice et al., 2018) or striking a target with a whip (Krotov et al., 2022; Nah et al., 2020). The study of such naturalistic interactions suggested that most of the conventional control objectives, such as maximizing smoothness or minimizing effort, are not sufficient to account for human behavior (Maurice et al., 2018; Nayeem et al., 2021). When investigating a task such as transporting a cup of coffee, more nuanced and object-centric objectives seem to be at play that are concerned with the stability and predictability of the interactions (Bazzi et al., 2018; Bazzi & Sternad, 2020b), and with minimizing the transient behavior of the object that is being controlled (Nayeem et al., 2021). Expanding on these insights, this study aims to understand complex interactions, with a focus on how embodiment affects higher-level control objectives, as well as the role of sensory feedback in the control process.

In interactions with the external world, physical properties of the body, especially the mechanical impedance of the limbs, have been identified as critical (Hogan, 1984, 2022). Due to its ability to store and dissipate mechanical energy, impedance affords stability in the presence of sensorimotor delays (Rancourt & Hogan, 2001) or when facing perturbations (Burdet et al., 2001). Impedance has been proposed as a dynamic primitive that interfaces the body with the external world and shapes basic patterns of movements (Hogan & Sternad, 2012, 2013). The controller determines the zero-force trajectories in interactions with the environment, and the observable movements arise as a consequence of this interaction (Hermus et al., 2020). From a physiological perspective, the central nervous system sets virtual or equilibrium-point trajectories via tuning of the phasic and tonic stretch reflexes (Feldman, 1986; Latash, 1992; Gomi & Kawato, 1997). However, to create rich behavior, the computational principles that generate more complex reference trajectories have remained elusive. This work sought to uncover how the inherent mechanics of the body—the “embodied intelligence”—and the characteristics of the neural controller influence one another in both the choice of control objectives and the use of sensory feedback.

In computational motor neuroscience, several lines of research have pursued the framework of optimal control as a model for the human motor controller (Todorov, 2004; Scott, 2004). The fundamental assumption in the optimal control framework is that the brain selects movements to optimize a certain objective or cost function. The nature of this objective function has been the focus of many studies, and several candidates have been proposed that are either based on kinematics (Flash & Hogan, 1985; Dingwell et al., 2004; Leib & Karniel, 2012; Svinin et al., 2019; Ronsse et al., 2010; Harris & Wolpert, 1998), kinetics (Todorov & Jordan, 2002b; Uno et al., 1989), or energetics (Anderson & Pandy, 2001; Wong et al., 2021). All of these objectives described human movements well in their respective experimental contexts. However, these costs may not be completely independent and free to choose by the brain. Wong et al. (2021) showed that kinematic smoothness arose as a by-product of physiological energy minimization, without any explicit kinematics-related cost function. This finding suggests that including more bio-fidelic features in the control model may achieve desired features in human behavior in a less top-down-dictated fashion. In this vein, mechanical impedance of the body, with its ability to store and dissipate energy, plays a critical role in defining the energetic landscape of the movements. Therefore, it is expected that the energy buffering of impedance affects properties of the controller, specifically the objective function.

The neural controller possesses feedback and feedforward control pathways that were recognized as early as 1899 (Woodworth, 1899). However, the exact interplay between feedforward and feedback control mechanisms and their dependence on different task scenarios is still under debate (Crevecoeur & Scott, 2014; Yeo et al., 2016). While an essential element for many accuracy-requiring tasks, for rapidly evolving physical interactions, such as carrying a cup filled with sloshing coffee, the sensorimotor information transmission is too slow to ensure successful feedback-based corrections. A more likely mechanism for instantaneous interaction dynamics is mechanical impedance as it affords rapid corrective responses without relying on neural signal transmission (Rancourt & Hogan, 2001; Abbs & Gracco, 1984). Indeed, previous results on interaction with the cup-and-ball system already highlighted that simple feedforward models that included impedance reproduced behavior well (Maurice et al., 2018; Nayeem et al., 2021). However, it remains unclear how feedback and feedforward processes interact with mechanical impedance in situations that require unplanned and rapid, yet complex, responses. This study applies expected and unexpected perturbations to examine the interplay of feedforward and feedback processes in interaction with impedance.

In overview, this study used the dynamically complex experimental task of transporting a cup of coffee to examine how interfacing the controller with the body's mechanical impedance affects the resulting behavior. To generate rich and informative data, the experiment included predictable and unpredictable perturbations in addition to the complex interaction forces generated through the object's internal dynamics. The optimal feedback control (OFC) framework was used as the model for the neural controller (Todorov & Jordan, 2002a). Simulations scrutinized how the inclusion of impedance in the model changed its behavior under different objective functions and involvement of sensory feedback. We hypothesized that mechanical impedance was a critical component to accurately generate interactions with the complex object (hypothesis 1), that the specifics of the objective function become less prominent due to the energy buffering of impedance (hypothesis 2), and that, for the same reasons, sensory feedback plays a subordinate role in this dynamically rich behavior (hypothesis 3).

Eleven healthy right-handed individuals participated in the experiment (19 to 25 years old, 8 females). None of the participants had any history of neurological disorders or biomechanical injuries in their upper limbs. Each subject provided written consent to the experimental procedures prior to participation. All subjects were given monetary compensation for their participation. The study was approved by the Institutional Review Board of Northeastern University.

This linearized model was used in both the virtual implementation for the human experiments, as well as in the simulations of different control models. While the linearized model no longer exhibited chaotic behavior, it still presented considerable challenges due to the underactuated degrees of freedom. A previous study demonstrated that the linearized version of the cup-and-ball system demanded slightly higher interaction forces as well as contained a more prohibitive antiresonance frequency, while other control challenges were similar to the nonlinear system (Sohn et al., 2020). Hence, the linearization was not expected to affect the purpose of this study.

Despite linearization, the virtual environment displayed the cup and ball as in previous studies; the angle of the ball in the cup corresponded to the linear state $ϕ$ in equation 2.2. As a control condition for the cup-and-ball system, the internal degree of freedom of the system (i.e., the ball) was constrained by fixing it to the bottom of the cup. This rigid-object condition essentially reduced the behavior to an unconstrained reaching movement with an increased hand mass. The inertia of this rigid object was matched to equate the sum of the cup and ball masses in the non-rigid system. The rigid object was displayed as an empty cup for visual consistency; however, its inertia matched the total mass of the cup and ball.

The visual information about the object movement and the virtual environment—the target boxes and the added perturbation bump (see details below)—were displayed on a vertical projection screen with a frame rate of 60 Hz. The total latency of the robotic interface and the visual projection, defined as the time difference between the moment the robot's handle moves until the time the movement is displayed on the screen, was 33 ms $±$ 17 ms. To measure this latency, a single video camera was used to simultaneously capture the robot's handle and the displayed cup position. This video was processed offline to digitize the positions of the robot and the cup, and the time lags between the two position signals were calculated by cross-correlation.

Participants stood in front of the projection screen and grasped the handle of the robotic manipulandum by which they could control the displayed object on the screen (see Figure 1B). They were instructed to move the object via the robot's handle along a horizontal line from a starting box (left) to a target box (right) displayed on the screen. The physical distance between the centers of start and target box was 32.5 cm. Each cup-and-ball trial started with the ball resting at the bottom of the cup. If the ball angle surpassed the rim of the cup during the movement, $ϕ>±45∘$⁠, the ball would “escape,” and the trial would be terminated as a failed attempt. The failed trials were not repeated. A successful trial ended when the cup was fully inside the target box and its speed fell below 5 mm/s. To avoid lengthy adjustments to home into the target, the start and target boxes were 30% wider than the width of the cup, allowing $±$5.3 mm tolerance. For the rigid-object condition, there were no failures; all trials were considered successful.

To promote consistency across subjects, participants were instructed via feedback to finish each trial within 1.4 s (⁠$±20%$⁠) for the rigid object and 1.8 s (⁠$±20%$⁠) for the cup-and-ball system. These movement times were chosen from preferred durations measured in pilot experiments. The movement durations were displayed at the end of each trial with the text color indicating whether the duration was within the acceptable bound (green), too fast (yellow), or too slow (red). This feedback was merely informative, and violation of the desired movement time did not lead to any explicit penalties or exclusion of the trial from analysis. Overall, subjects very quickly learned and adhered to the suggested movement time (see section 3.1).

In many trials, an external perturbation was applied that disturbed the movements (see Figure 1C). The perturbation was an impulsive force of magnitude $Fpert=20$ N, applied to the cup for 20 ms in the opposite direction of the movement, without any effect in the vertical direction. This perturbation was applied at a fixed location 60% into the movement distance toward the target. With the exception of the catch trials (see below), the existence of the perturbation in a trial was cued using a visual speed bump (see Figure 1C). Therefore, the participants were aware of its presence or absence before the trial began.

Each participant performed the task with both the rigid object and the cup-and-ball system in a total of 4 blocks of 100 trials each (see Figure 1D). The first 2 blocks involved performing the task with the rigid object, and blocks 3 and 4 presented the cup-and-ball system. Block 1 consisted of 95% unperturbed (null) trials and 5% perturbation trials (catch-perturbed), randomly interspersed across the block. Subjects were unaware of the perturbation prior to the trial onset, and there was no visual cue signaling the perturbation. Block 2 consisted of 95% perturbed trials (speed bump shown) and 5% null trials (catch-null), randomly interspersed across the block. Unlike the catch-perturbed trials in block 1, the catch-null trials in block 2 were also cued with a visually displayed speed bump on the way to the target like the remaining 95% of the trials in that block. Hence, subjects expected a perturbation, but no force was applied. Blocks 3 and 4 were the same as blocks 1 and 2, respectively, except that subjects interacted with the cup-and-ball system. The order of the blocks was the same for all participants. Overall, each participant performed 400 trials. Each block lasted approximately 10 to 12 minutes, with sufficient rest time given between the blocks. The experiment lasted approximately 1 hour from beginning to end.

A first analysis aimed to assess whether there were any improvements during each of the four blocks. This was particularly relevant in the cup-and-ball condition that posed more coordination challenges. Using movement time as an indicator, this was determined as follows: trial onset was defined as the time when the cup velocity exceeded 5 mm/s in direction of the target; the trial ended when the cup was fully inside the target box and its speed fell below 5 mm/s. To evaluate whether movement time changed over the course of practice, a Wilcoxon signed-rank test (within-subject, $N=11$⁠) compared the median trial duration of the first 25 and last 25 trials of each block. This nonparametric test was appropriate as the distributions of this variable did not meet the normality criterion.

To evaluate how subjects coped with the expected perturbations in the perturbed trials, the interaction forces were examined. Inspection of the trials in both the rigid-object and cup-and-ball trials showed that the force exhibited a characteristic discontinuity immediately following the perturbations (see section 3). To quantify this observed phenomenon in the different conditions, the force value at the onset of the perturbation was compared against its value immediately after the 20 ms interval of the perturbation. For each participant, the difference values of $Fpre-Fpost$ were tested against zero using unpaired $t$-tests. These tests were conducted separately for the rigid-object and the cup-and-ball conditions.

The cup velocity in the cup-and-ball condition rebounded after the perturbation. In the data, this rebound started immediately after the end of the perturbation. To quantify this feature, the cup's peak acceleration following the perturbation was identified in each trial. The peak values and the time of the peak relative to the end of the perturbation were analyzed.

The starting hypothesis was that arm impedance critically affected the model behavior, especially when additional external perturbations occurred. The two basic model structures compared the simulated behavior with both the rigid and the cup-and-ball object. These models included the most frequently used cost function that minimized effort (Todorov & Jordan, 2002b). The second point of interest was the effect of the cost function on behavior once impedance was included. To address this question, the two OFC models compared the behavior using the cost functions' minimum effort with minimum jerk (see Figures 2C and 2D, sections 2.2.4 and 2.2.5). The third analysis evaluated the role of sensory feedback, especially when facing unexpected external perturbations. This analysis focused on the catch trials and compared the basic feedback control model with a variant that eliminated feedback.

The perturbation force $Fpert$ was an impulse force that acted on the cup-and-ball system for a short time (20 ms) at the time the cup traveled 60% of the target distance. For null trials, $Fpert=0$ at all times. The numeric values of system parameters in the simulations were $M=3$ kg, $m=0.3$ kg, $l=0.5$ m, $g=9.81$ m/s$2$⁠, $G=5$⁠, and $τ=30$ ms. The hand mass $Marm$ was a free parameter in the model-fitting procedures (see section 2.3).

The cost term $Qt$ was defined as an $8×8$ diagonal matrix, with penalty terms considered for the cup position and velocity, and the ball angle, as shown in equation 2.8.

To study the effects of the objective function in our task, the objective of minimizing effort was contrasted with the objective of maximizing kinematic smoothness. Kinematic smoothness, frequently quantified by the time derivative of acceleration (jerk), is another widely discussed objective function for human movements (Flash & Hogan, 1985; Dingwell et al., 2004; Svinin et al., 2006, 2019). It must be noted that forces from the ball made the cup trajectory deviate from a pure minimum-jerk profile, i.e., it no longer exhibited a bell-shaped maximally smooth velocity profile (Bazzi & Sternad, 2020a). However, the OFC framework could include other cost terms, such as a penalty on the movement of the ball, to minimize jerk while taking into account the dynamics of the task.

As the control command $u$ in this OFC variant was directly defined as the jerk of cup movement, the penalty term $utTRtut$ in the cost function, equation 2.4, led to a minimum jerk trajectory, while taking into account the dynamics of the ball and the requirements of the task. For this model, the $Qt$ matrix was defined as a $5×5$ diagonal matrix with the same ball and cup penalty terms as in equation 2.8.

In both minimum-jerk variants (equations 2.12 and 2.14), the state penalties $Qt$ were defined similarly as in equation 2.8 with penalty terms for cup position and velocity, as well as the ball angle.

All four models described in equations 2.7, 2.10, 2.12, and 2.14, were developed based on the cup-and-ball system. To simulate the movements with the rigid object, the states regarding the ball movement (⁠$ϕ$ and $ϕ˙$⁠) were removed from the state vector, as well as the corresponding rows and columns from the matrices $A$⁠, $B$⁠, $H$⁠, and $Q$⁠.

The experimental design tested four different trial types: null, perturbation, catch-null, and catch-perturbed. For the perturbation trials and also the null trials, prior to the onset of movement, a visual cue (the speed bump or the absence of a bump) informed the subjects of the presence or absence of a perturbation. To incorporate this “knowledge” in the simulations, the perturbation force $Fpert$ was included as a state in the state-space equations; by initializing simulations with $Fpert=-20$ N, the knowledge about the time-dependent dynamics of the perturbation was already incorporated in the control system via the time-dependent $A$ matrix. For the null trials, the perturbation state $Fpert$ was set to zero. In this case, the perturbation was neither expected by the controller, nor did it occur.

For the feedforward simulations, the control command $u$ was first obtained from the optimal feedback control applied to a given nominal condition (e.g., for a null trial, Figure 3A), and then replayed in a feedforward manner. This controller was specifically used to simulate the catch conditions to assess how the control command without sensory feedback could compensate for unexpected perturbations.

Each control model contained free parameters that were used to fit the model predictions to experimental data. The free parameters for each model and their lower and upper bounds are summarized in Table 2. All other parameters, such as for the noise terms, were fixed across all models (see Table 1).

To fit each model, an offline nonlinear optimization algorithm searched for the parameter values that minimized the overall error of the fit (see below) across all four experimental blocks. The models were fit to the data of each participant separately. This optimization procedure consisted of two stages. In the first stage, a global optimization process was implemented to find appropriate initial values for the parameters to be used in a more refined optimization later. This stage was implemented using the particle swarm optimization algorithm (MATLAB function particleswarm, MATLAB 2021a, Mathworks Inc, Natick, MA) with a maximum of 300 iterations. In the second stage, the obtained parameters were used as a starting point for a nonlinear gradient-based optimization algorithm (sequential quadratic programming, implemented with fmincon in MATLAB 2021a), to converge to the optimal values faster.

For the rigid-object condition, $eϕ$ and $eϕ˙$ were removed.

The overall modeling error in equation 2.16 was used to statistically compare the selected models. The goal was to evaluate whether the model variants resulted in statistically different predictions quantified by the model-fitting errors.

To evaluate the effect of impedance on the model behavior (hypothesis 1), a three-way repeated-measures ANOVA was performed. The three factors were model type (with versus without impedance), object type (rigid object versus cup-and-ball), and perturbation type (null versus perturbed). The next analysis on the effects of the cost function applied a four-way repeated-measures ANOVA (hypothesis 2). The factors were model type, object type, and perturbation type as before, as well as the cost function (minimum-effort versus minimum-jerk). Catch trials were excluded for these two analyses. Finally, to assess the effects of feedback on performance, another four-way repeated-measures ANOVA was used with the factor feedback condition (feedback versus feedforward) instead of cost function (hypothesis 3). Only catch trials were used in the last analysis. To disentangle the interactions, post-hoc comparisons with Tukey-Kramer corrections were applied. All statistical tests were performed in MATLAB 2021a, using the ANOVA tools and multcompare function.

Participants were largely successful in all task conditions. When transporting the cup and ball in blocks 3 and 4, they lost the ball in only 56 trials out of the total of 2200 trials in this condition. The median success rates across participants (i.e., percentage of trials in which the ball was not lost) were 99% and 97% in blocks 3 and 4, respectively.

To probe whether there were improvements due to practice within each block, trial durations across the 100 trials of each block were examined. The median movement time between the first 25 and last 25 trials of each block were compared using the Wilcoxon signed rank test; failed trials and catch trials were excluded. The median movement time decreased with practice only in block 3 (unperturbed cup-and-ball condition; see Table 3). No significant changes in movement times were observed in other blocks. Note the significantly higher movement times in blocks 3 and 4 when the ball was sliding inside the cup; this observation gave first evidence for the higher demands when transporting the cup with the moving ball.

Prior to the onset of the predictable perturbations in blocks 2 and 4, participants' movement patterns were similar to those in the unperturbed trials, with both the rigid object and the cup-and-ball system (see Figure 4, darker brown trajectories). Until arriving at the perturbation location, participants continued to move with a velocity profile that showed a single prominent peak in the rigid object case and a plateau in the cup-and-ball condition (see Figures 4B and 4G). In both conditions, the impulsive resistive force caused a sudden drop in the cup velocity; in the cup-and-ball condition, the cup velocity rebounded after the perturbation (see Figures 4B and 4G), but this rebound was absent in the rigid-object condition. Further, with both objects, the interaction force showed a prominent spike (see Figures 4C and 4H). The interaction force exhibited a discontinuity at the time of perturbation as highlighted in the zoomed-in insets in Figures 4C and 4H; the interaction force right after the perturbation was consistently larger than its value at the onset of perturbation as the inset highlights. Averaged across participants, this increase was 0.99 $±$ 0.72 N with the rigid object (⁠$t(10)=4.553$⁠, $p=1.05×10-3$⁠) and 1.52$±$0.69 N with the cup-and-ball system (⁠$t(10)=7.323$⁠, $p=2.53×10-5$⁠).

The inclusion of the ball dynamics separated the two models' behavior further (see Figure 5B). Without impedance, the qualitative characteristics of the simulated movement deviated from the human data more distinctively. In contrast, with impedance, the OFC model modulated the interaction force and consequently the cup velocity less, leading to qualitatively similar behavior as seen in the human data. Similar to the rigid-object condition, the inclusion of impedance allowed the model to reproduce the discontinuous increase in interaction force from before to after the perturbation. Averaged across participants, the simulated interaction force increased by $0.81±0.54$ N (human data: $1.52±0.69$ N). This feature could not be captured in the absence of impedance, and the interaction force decreased by $-0.15±0.08$ N in the simulation. Another salient behavioral feature in the perturbed cup-and-ball condition was the smooth rebound of the cup velocity after the perturbation. In the face of perturbations, both models closely replicated the drop and rebound of the cup velocity as well as the spike in interaction force.

The overall error between the simulated and measured behavior was quantified using an aggregate root-mean-square error of fit that took into account the trajectories of all cup-and-ball states and the interaction force (see Figure 5C). In both null and perturbed conditions, this overall modeling error was smaller when impedance was included in the model. A three-way repeated-measures ANOVA revealed statistically significant two-way and three-way interactions and main effects. The full ANOVA results are presented in Table S-1. Because the goal was to compare the modeling error with and without impedance in each test condition, planned pair-wise comparisons were made within each block using Tukey-Kramer tests. Results indicated statistically significant differences between the two models in all four condition blocks (⁠$p=0.0016$⁠, $p=0.0038$⁠, $p=7.68×10-5$⁠, and $p=0.013$ for blocks 1 to 4, respectively). The inclusion of impedance resulted in significantly better fit to the data in all conditions.

For the above analyses, both models included minimization of effort as their objective function, consistent with many previous studies (Todorov & Jordan, 2002b; Diedrichsen et al., 2010). The following comparisons tested the influence of the objective function on the models' behavior in the context of interactions with a complex object.

In contrast, in the cup-and-ball condition, the two models without impedance produced distinct movement profiles (see Figure 7C). The minimum-jerk model variant generated a single-peaked velocity profile, as opposed to the pronounced double-peaked profile of the minimum-effort variant (compare the solid and dashed red lines in Figure 7C). However, when impedance was included, the two variants produced very similar movement patterns. Similar observations hold in the perturbed cup-and-ball condition (see Figure 7D): without impedance, the minimum-effort and minimum-jerk variants behaved distinctively from each other (solid and dashed red lines), while the two cost functions resulted in similar behavior when impedance was present in the model (solid and dashed blue lines).

These qualitative similarities and differences were clearly reflected in the quantitative error (see Figures 7E and 7F). Without impedance, the modeling errors from the minimum-effort and minimum-jerk variants were different in all four test conditions. However, with impedance, the two objective functions resulted in comparable modeling errors. A four-way repeated-measured ANOVA compared the modeling error with the factors model type (with versus without impedance), objective function (minimum-effort versus minimum-jerk), object type (rigid object versus cup-and-ball), and perturbation type (unperturbed versus perturbed). Results revealed that the model type produced a strong main effect (⁠$F(10)=1069.1$⁠, $p=1.69×10-11$⁠), while the cost function did not yield a significant main effect (⁠$F(10)=0.010$⁠, $p=0.920$⁠). Only the three-way and four-way interactions that involved model type and cost function showed statistical significance (⁠$F(10)≥17.13$⁠, $p≤0.002$⁠). The full ANOVA results are presented in Table S-2. Planned pairwise comparisons within each test condition (see Figures 7E to 7H) showed that the errors from the two model variants without impedance were different in all four blocks (⁠$p≤0.0132$⁠). However, with impedance, the two model variants were not statistically distinguishable in any of the blocks (⁠$p≥0.195$⁠).

To investigate the role of sensory feedback on the models' behavior, catch trials were analyzed. Note that catch trials were visually identical to the rest trials within the same block, and there was no indication that would inform the participants about the change in the perturbation condition. These trials afforded separating the effects of preplanning from a feedback-driven response in the face of the unexpected event.

The rebound in the cup velocity following the perturbation (see Figure 8C) distinguished the models even further. Without impedance, the velocity rebound deviated from the data even when the control model utilized sensory feedback (highlighted in the inset of Figure 8C). In the data, the cup acceleration peaked immediately (within 5 ms) after the perturbation ended. Across participants, the average cup acceleration before perturbation onset was $0.15±0.09$ m/s$2$ and rose to $1.06±0.27$ m/s$2$ immediately (within 5 ms) after the perturbation. This instantaneous behavior was observed in all participants. However, in the absence of impedance, the simulated cup acceleration did not exhibit this instantaneous rise; cup acceleration was $0.098±0.091$ m/s$2$ and $0.072±0.075$ m/s$2$⁠, before and after perturbation, respectively. In this case, the controller received sensory feedback with a 50 ms delay and could respond to the perturbation only after processing the delayed information. Consequently, the cup acceleration took $152±8$ ms to rise to a value that was comparable to the measured peak acceleration, within 1 standard deviation from the mean of the data. The model with impedance reproduced this instantaneous response after the perturbation, and cup acceleration peaked immediately following the perturbation and reached $0.71±0.30$ m/s$2$ (from $0.17±0.06$ m/s$2$ at perturbation onset). Note that although the feedforward model with impedance had an extra inertia parameter $Mref$ compared to the model with no impedance, this additional inertia cannot account for the rebound of cup velocity and acceleration after the perturbation (see Figure 8C). Such immediate rebound is achieved through a mechanism that can store and release energy in the face of a perturbation, which in our case is the stiffness element in the model.

To quantify and compare the error between different models, only the trajectories after the perturbation were taken into account because the feedback and feedforward responses were identical before the perturbation. As Figures 8E to 8H show, without impedance, the feedforward variant consistently underperformed the feedback-driven one in all four conditions. However, the errors from the feedback and feedforward variants of the models were comparable in the presence of impedance. A four-way repeated-measures ANOVA was used to analyze the modeling error; the four factors were model type (with versus without impedance), feedback structure (feedback versus feedforward), object type (rigid object versus cup-and-ball), and perturbation type (catch-null versus catch-perturbed). All two-, three- and four-way interactions that involved both model type and feedback structure revealed statistical significance (⁠$F(10)≥6.97$⁠, $p≤0.0247$⁠). Further, both model type and feedback structure showed strong main effects (⁠$F(10)≥99.1$⁠, $p≤1.66×10-6$⁠). The ANOVA results are shown in the supplementary materials. Following these results, planned pairwise comparisons of the model variants within each block (see Figures 8E to 8H) showed that the feedforward model without impedance underperformed the feedback one in all test conditions with statistical significance (⁠$p≤5.61×10-3$⁠). However, the two variants of the model with impedance were not statistically different in blocks 2, 3 and 4 (⁠$p≥0.084$⁠); only in block 1 did the feedback model perform better than the feedforward variant (⁠$p=5.80×10-4$⁠).

To date, numerous studies have examined the roles of the body mechanics, optimality principles, and sensory feedback for the control of voluntary goal-directed movements. Typically the experiments were carefully designed to isolate and explore the concept in question—for example, the cost functions (Todorov, 2004; Wolpert et al., 1995), mechanical impedance (Burdet et al., 2001; Selen et al., 2009), or the interaction between feedback and feedforward pathways (Maeda et al., 2020). The interplay of these critical elements, however, has received less attention, with only few exceptions. For instance, the interplay between sensory feedback and impedance was studied in an isometric holding task (Crevecoeur & Scott, 2014) and in a postural control task (Van Wouwe et al., 2022). Common to these studies are the simplified experimental tasks to generate clear data; for example, the feedforward command could safely be assumed constant in the cited articles. However, such simplified tasks may not be rich enough to reveal the full relation between the constituent elements of motor control. Hence, the goal of this work was to study the interplay of body mechanics, optimality criteria, and sensory feedback in an experimental assay in which humans interacted with an object that introduced internal dynamics with an underactuated degree of freedom. Such a movement task posed a new set of control challenges that afforded a more intricate view into control processes.

The body and its mechanics (embodiment) play a significant role in determining control, especially in more complex movements in interaction with objects and the environment. Thus, a model's prediction about the brain's role in motor control critically depends on the neuromechanical details included in the control model (Pinter et al., 2012). To address the crucial interplay of top-down control and body dynamics, this study examined OFC's predicted behavior when the model for the body was extended with a simple representation of its inherent dynamics, mechanical impedance. To evaluate the contribution of impedance, different variants of OFC were developed and compared.

Comparison of human kinematics in object interactions with the corresponding model simulations showed that the OFC model without impedance failed to reproduce several salient features in human performance. Several results highlighted the superior performance of the model that included impedance. First, the model with impedance produced kinematic profiles that deviated significantly less from the experimental trajectories, as quantified by the aggregate RMS error (see Figure 5C). Second, in the absence of impedance, the simulations could not reproduce the sudden increase in the interaction force immediately following the perturbation, a robust feature in the human data. Third, without impedance, the model's response showed a noticeable delay to an unexpected perturbation due to the 50 ms sensory delay in the model (see Figure 8C). No such delay was observed in the human data. However, when an impedance element was included, the model captured the subjects' overall movement profiles to a significantly greater degree. The impedance element acted as an energy buffer, allowing the model to capture the discontinuity in the interaction forces following a perturbation. Impedance also allowed the model to replicate the instantaneous response to unexpected perturbations.

The two phenomena at the perturbation, the change in interaction force and acceleration post-perturbation, occurred at a very fast timescale (⁠$<20$ ms). Neither voluntary nor involuntary neural responses could be generated at such timescales. For comparison, the fastest stretch reflex in upper extremities starts only 20 to 50 ms after perturbation onset (Pruszynski & Scott, 2012). These observations imply that the instantaneous responses should be attributed to the biomechanics of the arm. Within the 20 ms duration of the impulsive perturbation force, the cup velocity decreased significantly, while the cup position and also the position and velocity of the impedance's zero-force trajectory changed only negligibly. The drop in the cup's velocity caused a sudden increase in the damping force and, ultimately, an increase in the interaction force. The same instantaneous rise in interaction force was also the reason for the immediate rebound of cup velocity after the unexpected perturbation. The model without impedance lacked this mechanical contribution, and the velocity could accelerate and rebound only after processing the delayed feedback to command an increase in muscle force.

The impedance included in the model consisted of a pair of linear and constant spring and damper elements. Despite this simple design, the model clearly outperformed the one without impedance. In reality, nonlinear muscle, joint, and inter-limb mechanics contribute to the effective impedance of the limb, resulting in stiffness values that vary with posture (Gomi & Kawato, 1997; Perreault et al., 2001) or the evolving movement (Gomi & Kawato, 1996; Lee et al., 2012). Accurate representation of mechanical impedance is crucial to peel back the mechanics of the body and reveal the descending commands (Latash, 1994; Domen et al., 1999; Hogan, 2017; Hermus et al., 2020). In addition to the passive contributions of the musculoskeletal properties, task-dependent modulation of stiffness has also been proposed as a means of control (Burdet et al., 2001, 2006; Franklin et al., 2007). The effects of time-varying modulation of impedance have been studied in musculoskeletal simulations (Van Wouwe et al., 2022; Xiong et al., 2021; Berret & Jean, 2020; Shourijeh & Fregly, 2020) and in robot control (Hammoud et al., 2021). Results suggested that the brain may indeed use impedance-modulation as another control mechanism. However, it remains a challenge to parse the contributions of active stiffness modulation from the variations caused by the passive mechanics of the body due to the persistent challenges of estimating time-varying impedance in vivo (Rouse et al., 2014). To avoid unnecessary complications and assumptions, the simple time-invariant model was adopted that still describes the human data remarkably well.

The behavior of the models critically depended on the parameter values used in the simulations. Although the models were highly simplified compared to the human physiology, the lumped parameter values can be discussed in relation to biomechanical properties of the body. The identified impedance parameters were within the range of previously reported values for the end-point (hand) impedance, although the range reported in the literature spans one order of magnitude. The estimated end-point stiffness in the literature ranged from 40 N/m (Nayeem et al., 2020; Nayeem et al., 2023) to 200 N/m (Maurice et al., 2018) during rhythmic movements, and $>250$ N/m (Perreault et al., 2001; Krutky et al., 2013) and $>60$ N/m (Mussa-Ivaldi et al., 1985) in isometric holding tasks where short-range stiffness was a major contributor. In comparison, the end-point stiffness identified via matching the model to the experimental data in our work was $49±12$ N/m and $59±47$ N/m in the minimum-effort and minimum-jerk models, respectively. Likewise, our estimated damping was $9.3±3.4$ N.s/m and $18±3$ N.s/m in the two models, which were within the range of $10-50$ N.s/m reported in (Nayeem et al., 2020; Nayeem et al., 2023; Maurice et al., 2018; Krutky et al., 2013).

In the models that included impedance, the mass of the arm $Marm$ represented the effective inertia after the impedance element; it was estimated to be $0.78±0.23$ kg (minimum-effort) and $0.61±0.14$ kg (minimum-jerk), which is slightly higher than the mass of the hand—on average 0.5% to 0.6% of body mass (Dumas et al., 2007), approximately 0.3 to 0.55 kg. This consistency can be explained by noting that forearm and upper arm partly contributed to the effective post-impedance inertia. In the absence of impedance, the identified $Marm$ was $3.3±2.0$ kg, which was close to the total mass of the arm—4.2% to 4.7% of body mass or 2.5 to 4.3 kg (Dumas et al., 2007). The lumped parameter $Mref=17±2$ kg obtained when fitting the data was much larger than that of the arm. This could be physiologically realistic because this parameter represented the effective inertia of all force-producing machinery in the body. This effective inertia might include the contributions of trunk and even leg muscles, as it is not unreasonable to assume that core muscles also engage in responding to perturbations, hence, leading to a relatively larger inertia value for $Mref$⁠.

Not only did the added mechanical impedance replicate human movements much better; it also provided further important insights about two cornerstones of optimal feedback control: detailed features of the cost function became less important when impedance was added, and sensory feedback for the processing of error and correction became less critical. These two issues are discussed next.

To answer why a certain movement pattern is chosen over possible alternatives, the cornerstone of optimal control theory is that the controller, or the brain, seeks to minimize a certain objective function. In movement neuroscience, there has been considerable debate as to which objective function the brain is trying to optimize (see Todorov, 2004, for a review). A multitude of objective functions have been proposed, ranging from the kinematics-based objectives, for example, minimum jerk (Flash & Hogan, 1985; Svinin et al., 2019), minimum crackle (Dingwell et al., 2004), and minimum acceleration (Leib & Karniel, 2012), to the kinetics-based ones such as minimum torque change (Uno et al., 1989) and minimum muscle effort (Todorov & Jordan, 2002b; Ronsse et al., 2010), to the minimum variance cost (Harris & Wolpert, 1998) and the physiological energy expenditure (Anderson & Pandy, 2001; Wong et al., 2021). All of these objectives were able to describe human movements reasonably well in their respective experimental contexts. Support for any of these objectives typically involved comparing them against other objectives for a given task and then choosing the one that better matched the data. In the present study, the comparisons made between the two OFC variants with two different objective functions, minimum effort and jerk, also displayed distinct differences in the movement patterns. Strikingly, though, when impedance was included in the controller, no quantifiable differences were observed, and both cost functions replicated the human behavior equally well. This finding suggests that when OFC included even minimal biomechanical features, details of the objective function may become inconsequential. In the same spirit, Diedrichsen and colleagues (2010) have shown that under the assumption of signal-dependent noise, the objective of reducing effort is equivalent to reducing end-point variance. Similarly, Wong et al. (2021) showed that smooth movements are more energy efficient. Here, it is shown that the physiologically realistic element of arm impedance masks distinctions between effort and smoothness. One important implication is that the underlying control principles of the brain may be fundamentally unobservable due to the filtering effects of the body mechanics. However, it needs to be kept in mind that this finding was obtained in the context of interactions with a complex object. To what degree this blurring of details holds for other tasks requires more investigation.

Features of feedforward (preplanned) and feedback (online corrections) control mechanisms are intertwined in coordinated human movement. Following Woodworth in 1899, numerous authors have distinguished a ballistic (feedforward) and a homing-in phase associated with error corrections (feedback) in simple pointing movements (Woodworth, 1899; Meyer et al., 1988). In more recent years, Crevecoeur & Scott (2014) argued that feedback control was necessary to complement a feedforward command to replicate the human response to perturbations in an isometric task. Yeo et al. (2016) showed that when sensory information was uncertain, human movements could not be explained by feedback control alone and feedforward pathways were necessary. Even in tasks that heavily rely on continuous sensory information, such as balancing a stick on the fingertip, there are compelling arguments for intermittent episodes of feedforward predictive control (Loram et al., 2012).

This study used a dynamically complex task where the underactuated object created perturbations depending on the hand's actions. In addition to expected perturbations, unexpected perturbations in the catch trials were given to identify the role of sensory feedback. The simulations of the catch trials without an impedance element showed that the feedforward replay of the OFC's control command was unable to cope with the perturbations or their unexpected absence. This behavior did not match the subjects' behavior (see Figure 8). On the other hand, the feedback-driven OFC model achieved a better fit to the data with significantly smaller model error. Based on these results, one may readily conclude that a feedback loop is necessary. However, when the model included impedance, both the feedback and feedforward variants replicated the experimental data equally well. These findings showed that mechanical effects may have intricate contributions to coping with complex events. Even a task that involves underactuated dynamics and challenging requirements, such as responding to the disturbance without “spilling the coffee” may be achieved by a feedforward control scheme as long as it is mediated by body mechanics.

It must be noted that our results do not simply provide support for either feedforward or feedback control. Instead, the key insight from these results is that under biomechanically more plausible assumptions, the distinction between predictive feedforward and sensory-driven feedback control may become less apparent. Similar effects were reported previously in a simple reaching task by Berret and Jean (2020), who showed that attenuation of task-dependent errors could be achieved by a feedforward tuning of the arm impedance. It should also be noted that these results must be interpreted in the context of the chosen experimental task. It is obvious that large enough and long enough perturbations can disrupt the movement to a degree that requires sensory-driven corrections in terms of both long-latency reflexes and voluntary responses. Indeed, Crevecoeur and Scott (2014) showed that optimal feedback corrections were necessary to complement mechanical impedance after large perturbations in a holding task. However, given that arm impedance is nonlinear (for example, it exhibits short-range stiffness; Joyce et al., 1969; Rack & Westbury, 1974) and that it changes with postural configuration (Pando et al., 2014) and movement (Gomi & Kawato, 1996; Lee et al., 2012), great care is needed when searching for the control structure using perturbations. New computational techniques that simultaneously model feedback and feedforward pathways as developed in a recent study may provide new insights (Van Wouwe et al., 2022).

From a mathematical perspective, the impedance element in the model introduced three additional states to the dynamics of the system that the optimal feedback controller had to address. However, unlike inertial mechanics that can also increase the number of states (e.g., in multi-segment arm models), impedance endows the system with potential energy storage and dissipation properties. These properties could not be achieved with inertial mechanics alone. The implications of this physical and mathematical change in the system can be interpreted from two perspectives.

A first position puts priority to the fact that the brain is the site of control and the body is the system to be controlled. Thus, the brain as the sole control authority is fully responsible for defining the characteristics of the movements based on knowledge about the body. Grounded in this dualistic viewpoint, much of neuroscience has tried to shed light on the neural representations and neural activity patterns for the generation of movements. The search for the “control signals” in the primary motor or parietal cortices that directly correlate with the behavior at the single-neuron (Georgopoulos et al., 1986) or population level (Churchland et al., 2012; Shenoy et al., 2013) follows this basic position, if not philosophically, then at least practically. To facilitate this search, the movements themselves have been held simple to allow many repetitions under the “same” conditions and also to eliminate any interactive effects from the environment. Many studies on intracortical data in nonhuman primates have adopted such simple center-out reaching (Georgopoulos et al., 1986; Churchland et al., 2012) or cranking tasks (Russo et al., 2020). While intriguing insights have been gained, interactions with an object and the environment would present a step up in complexity and in conceptual terms. Our focus on the interaction with a dynamical object indicates that the physical properties of the body may present important challenges to the interpretability of purely top-down perspectives. Concretely, following our results, issues whether incoming sensory information affected the behavior and whether the cost function was effort or smoothness became blurred by adding simple biomechanical elements. In light of these findings it becomes evident that specific facets of control might be filtered out by the dynamics of the body.

In a second view, innate dynamical properties of the body, such as mechanical impedance, are interpreted as a lower-level autonomous control authority that shares the control responsibility with the higher-level information processors. Thus, this embodiment of control means that not all characteristics of observable behavior can be attributed to the higher-level controller. The significance of the innate passive dynamics for producing movements was strikingly demonstrated in a series of passive walkers (McGeer, 1990; Collins et al., 2005). These non-actuated mechanical linkages could produce stable gaits in the absence of any motors, sensors, or a “brain.” Passive interactions of the body with a complex environment could even allow a dead fish to swim upstream (Beal et al., 2006).

Undeniably, functional human behavior is richer than these passive movements, and bodily mechanics alone cannot generate the full spectrum of functionally specific behavior. Hence, descending neural commands need to be sensitively interfaced to not only account for, but also leverage bodily mechanics. The physical properties of the body provide useful resources for embodied computation for both perception and action. Morphological computation for guided actions is well studied in animals. For instance, a housefly's compound eyes can afford complex navigation skills using simple visuomotor circuitry (Franceschini et al., 1992), and the biomechanics of a goat's hoof enables localized slip control in rough terrains (Abad et al., 2016). Similarly in humans, it has been demonstrated that joint compliance plays a significant role in accurate haptic perception: correct tuning of stiffness largely simplifies the computational load during control of movement (Sornkarn et al., 2016). In our results, mechanical impedance of the arm proved capable of negotiating the unexpected perturbations in a feedback-deprived control scheme, highlighting the possibility of producing “intelligent” responses through morphological or embodied computation.

Further, physical processes on multiple scales, ranging from passive muscle force enhancement via titin proteins (Herzog, 2018) to limb inertial mechanics, provide means to distribute control authority across the body, thereby alleviating some of the computational burden on the brain. In human movement neuroscience, only a few approaches have placed emphasis on the dynamics emerging from the interplay between control and the dynamics of the body. From a neurophysiological perspective, the equilibrium point hypothesis aimed to reveal how the central nervous system specifies lambda, the equilibrium or reference configuration of a limb, via tuning of the phasic and tonic stretch reflexes (Feldman, 1986). From a mechanical perspective, a number of studies have demonstrated the tuning of impedance during movement (Burdet et al., 2001; Franklin et al., 2007). Further, studies on bimanual coordination have detailed how rhythmically moving limbs self-organize into autonomous nonlinear oscillators that synchronize into stable phase relations without requiring detailed top-down control (Sternad et al., 1998; Ronsse et al., 2009). Hence, movements may be the emerging response of the body leveraging the environment with simple parameter specifications or motor commands (Sternad et al., 2000). However, to date, these approaches have been limited to simple tasks. In order to advance one step further, this study employed optimal feedback control as a high-level controller that guides the behavior of the low-level dynamics as a way to reconcile the duality between information processing in the brain and embodiment. Such interfacing ultimately shapes the final behavior.

This study highlighted the importance of mechanical impedance of the body in modeling human movements. The experimental assay was humans transporting a complex object with underactuated dynamics. Even the simple linear transport revealed that the unactuated degree of freedom required more than a bell-shaped velocity profile of the hand, quintessential in reaching or transporting a rigid body. Without a regard for the compliance of the body, an information processing OFC model fell short of reproducing characteristic features of human movements; a successful optimal feedback controller necessitated embodiment (mechanical impedance) in the control loop. Perturbation trials further revealed that impedance buffered the energy and thereby obviated the need for feedback in the optimal controller. Including impedance into the model also blurred the distinction between cost functions previously suggested for various reaching tasks. Taken together, our results emphasize that attention must be given to body mechanics to better understand the controller. The results highlight the formidable challenge to gain insights into the neural controller due to its tight intertwining with the body's and the task's dynamics.

The research was funded by the National Science Foundation, NSF-NRI-1637854, NSF-M3X-1825942, and the National Institutes of Health, NIH-R37-HD087089 and NIH-R01-CRCNS-NS120579 (awarded to D.S.). R.S.R. was also funded by NSERC Postdoctoral Fellowship PDF-532835-2019. We also thank Piet Lammertse for continued technical assistance and Frédéric Crevecoeur (Crevecoeur et al., 2019) for providing the open-source codes for OFC implementation in MATLAB.