To effectively behave within ever-changing environments, biological agents must learn and act at varying hierarchical levels such that a complex task may be broken down into more tractable subtasks. Hierarchical reinforcement learning (HRL) is a computational framework that provides an understanding of this process by combining sequential actions into one temporally extended unit called an option. However, there are still open questions within the HRL framework, including how options are formed and how HRL mechanisms might be realized within the brain. In this review, we propose that the existing human motor sequence literature can aid in understanding both of these questions. We give specific emphasis to visuomotor sequence learning tasks such as the discrete sequence production task and the M × N (M steps × N sets) task to understand how hierarchical learning and behavior manifest across sequential action tasks as well as how the dorsal cortical–subcortical circuitry could support this kind of behavior. This review highlights how motor chunks within a motor sequence can function as HRL options. Furthermore, we aim to merge findings from motor sequence literature with reinforcement learning perspectives to inform experimental design in each respective subfield.

The mismatch negativity (MMN) is an ERP component seen in response to unexpected “novel” stimuli, such as in an auditory oddball task. The MMN is of wide interest and application, but the neural responses that generate it are poorly understood. This is in part due to differences in design and focus between animal and human oddball paradigms. For example, one of the main explanatory models, the “predictive error hypothesis”, posits differences in timing and selectivity between signals carried in auditory and prefrontal cortex (PFC). However, these predictions have not been fully tested because (1) noninvasive techniques used in humans lack the combined spatial and temporal precision necessary for these comparisons and (2) single-neuron studies in animal models, which combine necessary spatial and temporal precision, have not focused on higher order contributions to novelty signals. In addition, accounts of the MMN traditionally do not address contributions from subcortical areas known to be involved in novelty detection, such as the amygdala. To better constrain hypotheses and to address methodological gaps between human and animal studies, we recorded single neuron activity from the auditory cortex, dorsolateral PFC, and basolateral amygdala of two macaque monkeys during an auditory oddball paradigm modeled after that used in humans. Consistent with predictions of the predictive error hypothesis, novelty signals in PFC were generally later than in auditory cortex and were abstracted from stimulus-specific effects seen in auditory cortex. However, we found signals in amygdala that were comparable in magnitude and timing to those in PFC, and both prefrontal and amygdala signals were generally much weaker than those in auditory cortex. These observations place useful quantitative constraints on putative generators of the auditory oddball-based MMN and additionally indicate that there are subcortical areas, such as the amygdala, that may be involved in novelty detection in an auditory oddball paradigm.

Real-world decision-making often involves social considerations. Consequently, the social value of stimuli can induce preferences in choice behavior. However, it is unknown how financial and social values are integrated in the brain. Here, we investigated how smiling and angry face stimuli interacted with financial reward feedback in a stochastically rewarded decision-making task. Subjects reliably preferred the smiling faces despite equivalent reward feedback, demonstrating a socially driven bias. We fit a Bayesian reinforcement learning model to factor the effects of financial rewards and emotion preferences in individual subjects, and regressed model predictions on the trial-by-trial fMRI signal. Activity in the subcallosal cingulate and the ventral striatum, both involved in reward learning, correlated with financial reward feedback, whereas the differential contribution of social value activated dorsal temporo-parietal junction and dorsal anterior cingulate cortex, previously proposed as components of a mentalizing network. We conclude that the impact of social stimuli on value-based decision processes is mediated by effects in brain regions partially separable from classical reward circuitry.

Although much is known about decision making under uncertainty when only a single step is required in the decision process, less is known about sequential decision making. We carried out a stochastic sequence learning task in which subjects had to use noisy feedback to learn sequences of button presses. We compared flat and hierarchical behavioral models and found that although both models predicted the choices of the group of subjects equally well, only the hierarchical model correlated significantly with learning-related changes in the magneto-encephalographic response. The significant modulations in the magneto-encephalographic signal occurred 83 msec before button press and 67 msec after button press. We also localized the sources of these effects and found that the early effect localized to the insula, whereas the late effect localized to the premotor cortex.

We sought to determine how a visual maze is mentally solved. Human subjects ( N = 13) viewed mazes with orthogonal, unbranched paths; each subject solved 200-600 mazes in any specific experiment below. There were four to six openings at the perimeter of the maze, of which four were labeled: one was the entry point and the remainder were potential exits marked by Arabic numerals. Starting at the entry point, in some mazes the path exited, whereas in others it terminated within the maze. Subjects were required to type the number corresponding to the true exit (if the path exited) or type zero (if the path did not exit). In all cases, the only required hand movement was a key press, and thus the hand never physically traveled through the maze. Response times (RT) were recorded and analyzed using a multiple linear regression model. RT increased as a function of key parameters of the maze, namely the length of the main path, the number of turns in the path, the direct distance from entry to termination, and the presence of an exit. The dependence of RT on the number of turns was present even when the path length was fixed in a separate experiment ( N = 10 subjects). In a different experiment, subjects solved large and small mazes ( N = 3 subjects). The former was the same as the latter but was scaled up by 1.77 times. Thus both kinds of mazes contained the same number of squares but each square subtended 1.77° of visual angle (DVA) in the large maze, as compared to 1 DVA in the small one. We found that the average RT was practically the same in both cases. A multiple regression analysis revealed that the processing coefficients related to maze distance (i.e., path length and direct distance) were reduced by approximately one-half when solving large mazes, as compared to solving small mazes. This means that the efficiency in processing distance-related information almost doubled for scaled-up mazes. In contrast, the processing coefficients for number of turns and exit status were practically the same in the two cases. Finally, the eye movements of three subjects were recorded during maze solution. They consisted of sequences of saccades and fixations. The number of fixations in a trial increased as a linear function of the path length and number of turns. With respect to the fixations themselves, eyes tended to fixate on the main path and to follow it along its course, such that fixations occurring later in time were positioned at progressively longer distances from the entry point. Furthermore, the time the eyes spent at each fixation point increased as a linear function of the length and number of turns in the path segment between the current and the upcoming fixation points. These findings suggest that the maze segment from the current fixation spot to the next is being processed during the fixation time (FT), and that a significant aspect of this processing relates to the length and turns in that segment. We interpreted these relations to mean that the maze was mentally traversed. We then estimated the distance and endpoint of the path mentally traversed within a specific FT; we also hypothesized that the next portion of the main path would be traversed during the ensuing FT, and so on for the whole path. A prediction of this hypothesis is that the upcoming saccade would land the eyes at or near the locus on the path where the mental traversing ended, so that “the eyes would pick up where the mental traversal left off.” In this way, a portion of the path would be traversed during a fixation and successive such portions would be strung together closely along the main path to complete the processing of the whole path. We tested this prediction by analyzing the relations between the path distance of mental traverse and the distance along the path between the current and the next fixation spot. Indeed, we found that these distances were practically the same and that the endpoint of the hypothesized mental path traversing was very close to the point where the eye landed by the saccade to initiate a new mental traversing. This forward progression of fixation points along the maze path, coupled with the ongoing analysis of the path between successive fixation points, would constitute an algorithm for the routine solution of a maze.