Exploration is an important part of decision making and is crucial to maximizing long-term rewards. Past work has shown that people use different forms of uncertainty to guide exploration. In this study, we investigate the role of the pupil-linked arousal system in uncertainty-guided exploration. We measured participants' ( n = 48) pupil dilation while they performed a two-armed bandit task. Consistent with previous work, we found that people adopted a hybrid of directed, random, and undirected exploration, which are sensitive to relative uncertainty, total uncertainty, and value difference between options, respectively. We also found a positive correlation between pupil size and total uncertainty. Furthermore, augmenting the choice model with subject-specific total uncertainty estimates decoded from the pupil size improved predictions of held-out choices, suggesting that people used the uncertainty estimate encoded in pupil size to decide which option to explore. Together, the data shed light on the computations underlying uncertainty-driven exploration. Under the assumption that pupil size reflects locus coeruleus-norepinephrine neuromodulatory activity, these results also extend the theory of the locus coeruleus-norepinephrine function in exploration, highlighting its selective role in driving uncertainty-guided random exploration.

To survive in complex environments, animals need to have mechanisms to select effective actions quickly, with minimal computational costs. As perhaps the computationally most parsimonious of these systems, Pavlovian control accomplishes this by hardwiring specific stereotyped responses to certain classes of stimuli. It is well documented that appetitive cues initiate a Pavlovian bias toward vigorous approach; however, Pavlovian responses to aversive stimuli are less well understood. Gaining a deeper understanding of aversive Pavlovian responses, such as active avoidance, is important given the critical role these behaviors play in several psychiatric conditions. The goal of the current study was to establish a behavioral and computational framework to examine aversive Pavlovian responses (activation vs. inhibition) depending on the proximity of an aversive state (escape vs. avoidance). We introduce a novel task in which participants are exposed to primary aversive (noise) stimuli and characterized behavior using a novel generative computational model. This model combines reinforcement learning and drift-diffusion models so as to capture effects of invigoration/inhibition in both explicit choice behavior as well as changes in RT. Choice and RT results both suggest that escape is associated with a bias for vigorous action, whereas avoidance is associated with behavioral inhibition. These results lay a foundation for future work seeking insights into typical and atypical aversive Pavlovian responses involved in psychiatric disorders, allowing us to quantify both implicit and explicit indices of vigorous choice behavior in the context of aversion.

Decision-making algorithms face a basic tradeoff between accuracy and effort (i.e., computational demands). It is widely agreed that humans can choose between multiple decision-making processes that embody different solutions to this tradeoff: Some are computationally cheap but inaccurate, whereas others are computationally expensive but accurate. Recent progress in understanding this tradeoff has been catalyzed by formalizing it in terms of model-free (i.e., habitual) versus model-based (i.e., planning) approaches to reinforcement learning. Intuitively, if two tasks offer the same rewards for accuracy but one of them is much more demanding, we might expect people to rely on habit more in the difficult task: Devoting significant computation to achieve slight marginal accuracy gains would not be “worth it.” We test and verify this prediction in a sequential reinforcement learning task. Because our paradigm is amenable to formal analysis, it contributes to the development of a computational model of how people balance the costs and benefits of different decision-making processes in a task-specific manner; in other words, how we decide when hard thinking is worth it.

Imagination enables us not only to transcend reality but also to learn about it. In the context of reinforcement learning, an agent can rationally update its value estimates by simulating an internal model of the environment, provided that the model is accurate. In a series of sequential decision-making experiments, we investigated the impact of imaginative simulation on subsequent decisions. We found that imagination can cause people to pursue imagined paths, even when these paths are suboptimal. This bias is systematically related to participants' optimism about how much reward they expect to receive along imagined paths; providing feedback strongly attenuates the effect. The imagination effect can be captured by a reinforcement learning model that includes a bonus added onto imagined rewards. Using fMRI, we show that a network of regions associated with valuation is predictive of the imagination effect. These results suggest that imagination, although a powerful tool for learning, is also susceptible to motivational biases.

We examined the maturation of decision-making from early adolescence to mid-adulthood using fMRI of a variant of the Iowa gambling task. We have previously shown that performance in this task relies on sensitivity to accumulating negative outcomes in ventromedial PFC and dorsolateral PFC. Here, we further formalize outcome evaluation (as driven by prediction errors [PE], using a reinforcement learning model) and examine its development. Task performance improved significantly during adolescence, stabilizing in adulthood. Performance relied on greater impact of negative compared with positive PEs, the relative impact of which matured from adolescence into adulthood. Adolescents also showed increased exploratory behavior, expressed as a propensity to shift responding between options independently of outcome quality, whereas adults showed no systematic shifting patterns. The correlation between PE representation and improved performance strengthened with age for activation in ventral and dorsal PFC, ventral striatum, and temporal and parietal cortices. There was a medial-lateral distinction in the prefrontal substrates of effective PE utilization between adults and adolescents: Increased utilization of negative PEs, a hallmark of successful performance in the task, was associated with increased activation in ventromedial PFC in adults, but decreased activation in ventrolateral PFC and striatum in adolescents. These results suggest that adults and adolescents engage qualitatively distinct neural and psychological processes during decision-making, the development of which is not exclusively dependent on reward-processing maturation.