This letter proposes a new idea to improve learning efficiency in reinforcement learning (RL) with the actor-critic method used as a muscle controller for posture stabilization of the human arm. Actor-critic RL (ACRL) is used for simulations to realize posture controls in humans or robots using muscle tension control. However, it requires very high computational costs to acquire a better muscle control policy for desirable postures. For efficient ACRL, we focused on embodiment that is supposed to potentially achieve efficient controls in research fields of artificial intelligence or robotics. According to the neurophysiology of motion control obtained from experimental studies using animals or humans, the pedunculopontine tegmental nucleus (PPTn) induces muscle tone suppression, and the midbrain locomotor region (MLR) induces muscle tone promotion. PPTn and MLR modulate the activation levels of mutually antagonizing muscles such as flexors and extensors in a process through which control signals are translated from the substantia nigra reticulata to the brain stem. Therefore, we hypothesized that the PPTn and MLR could control muscle tone, that is, the maximum values of activation levels of mutually antagonizing muscles using different sigmoidal functions for each muscle; then we introduced antagonism function models (AFMs) of PPTn and MLR for individual muscles, incorporating the hypothesis into the process to determine the activation level of each muscle based on the output of the actor in ACRL. ACRL with AFMs representing the embodiment of muscle tone successfully achieved posture stabilization in five joint motions of the right arm of a human adult male under gravity in predetermined target angles at an earlier period of learning than the learning methods without AFMs. The results obtained from this study suggest that the introduction of embodiment of muscle tone can enhance learning efficiency in posture stabilization disorders of humans or humanoid robots.

Humans are able to robustly maintain desired motion and posture under dynamically changing circumstances, including novel conditions. To accomplish this, the brain needs to optimize the synergistic control between muscles against external dynamic factors. However, previous related studies have usually simplified the control of multiple muscles using two opposing muscles, which are minimum actuators to simulate linear feedback control. As a result, they have been unable to analyze how muscle synergy contributes to motion control robustness in a biological system. To address this issue, we considered a new muscle synergy concept used to optimize the synergy between muscle units against external dynamic conditions, including novel conditions. We propose that two main muscle control policies synergistically control muscle units to maintain the desired motion against external dynamic conditions. Our assumption is based on biological evidence regarding the control of multiple muscles via the corticospinal tract. One of the policies is the group control policy (GCP), which is used to control muscle group units classified based on functional similarities in joint control. This policy is used to effectively resist external dynamic circumstances, such as disturbances. The individual control policy (ICP) assists the GCP in precisely controlling motion by controlling individual muscle units. To validate this hypothesis, we simulated the reinforcement of the synergistic actions of the two control policies during the reinforcement learning of feedback motion control. Using this learning paradigm, the two control policies were synergistically combined to result in robust feedback control under novel transient and sustained disturbances that did not involve learning. Further, by comparing our data to experimental data generated by human subjects under the same conditions as those of the simulation, we showed that the proposed synergy concept may be used to analyze muscle synergy–driven motion control robustness in humans.