We examined dynamic interactions between cortex and BG during stimulus–response and feedback processing phases of categorization. First, we dissociated stimulus–response processing from feedback processing using “jittered” intervals of time between response and feedback to examine how each recruits the four primary corticostriatal loops (motor, executive, visual, and motivational). Second, we examined dynamic interactions within and between corticostriatal loops using Granger causality mapping. On each trial, subjects viewed one of six abstract visual stimuli, pressed a button indicating category membership, and then received feedback as to whether the decision was right or wrong. Stimulus–response processing was associated with greater activity in the visual loop, whereas feedback processing resulted in activity in the executive loop that was sensitive to feedback valence. Granger causality mapping showed patterns of directed influence within corticostriatal loops and between loops from the motor to the executive, to the visual, and finally to the motivational loop. These patterns of interaction are consistent with functional integration of motor processing in the motor loop with feedback processing in the executive loop and maintenance of stimulus–response history for future responses in the motivational loop.
Categorization is the process of learning to distinguish between different types or groups of objects and situations in the world. Previous studies have identified the BG as critical for category learning in both human and nonhuman animals (Seger & Miller, 2010; Shohamy, Myers, Kalanithi, & Gluck, 2008; Poldrack et al., 2001; Poldrack, Prabhakaran, Seger, & Gabrieli, 1999; Knowlton, Mangels, & Squire, 1996). Along with the BG, various cortical areas have been implicated in category learning (Fellows & Farah, 2003; Cools, Clark, Owen, & Robbins, 2002; Monchi, Petrides, Petre, Worsley, & Dagher, 2001; Lombardi et al., 1999). Our focus here is on examining the functions of corticostriatal systems, with the goal of understanding the division of labor across individual cortical and BG regions and how these regions interact to support learning.
Stimulus–Response and Feedback Processing in the Corticostriatal Loops
Typical trial and error visual categorization learning tasks can be divided into two phases: the stimulus–response phase and the feedback processing phase. In the former, the subject views a stimulus, decides on its category membership, and executes the appropriate motor response indicating the category membership of the stimulus. In the latter, subjects receive feedback indicating whether their response was correct or incorrect and use this information to update their knowledge of the correct category membership. The first goal of the present study is to dissociate neural activity associated with stimulus processing and response selection from that associated with feedback processing within the same task by manipulating the temporal delay between these phases and extend previous work by directly comparing these separate events. This approach was similar to that used by Aron and colleagues (2004) as well as Daniel and Pollmann (2010); however, neither study reported direct comparisons between stimulus–response and feedback phases. This study extends this previous work by directly comparing these separate events, with an emphasis on striatal activity. More importantly, we use this approach of separating stimulus and feedback events to better examine how the cortex and striatum work in parallel to subserve category learning.
Anatomically, the striatum interacts with cortex and other BG regions via recurrent networks referred to as “corticostriatal loops.” Cortical regions project to the striatum, which in turn project to the output nuclei of the BG, and then to the thalamus before returning to cortex. We divide the corticostriatal system into four loops: motivational, executive, visual, and motor (Seger, 2008; Lawrence, Sahakian, & Robbins, 1998). However, it should be noted that these loops overlap and there are no discrete boundaries between loops (Alexander, DeLong, & Strick, 1986).
During category learning, we have proposed that the visual loop helps select the appropriate visual representations of stimuli and map them to appropriate behaviors (Seger, 2008). The visual loop is comprised of cortical regions that underlie visual processing such as the extrastriate occipital lobe, inferior temporal cortex, middle temporal cortex, and FEFs. These cortical regions project to the posterior part of the caudate nucleus. The motor loop is thought to facilitate the selection and execution of appropriate motor responses depending on the task demands (Seger, 2008). The motor loop consists of the putamen interacting with primary motor cortex, somatosensory cortex, and motor planning areas such as the premotor and supplementary motor cortices. Both the motor and visual corticostriatal loops are active during categorization learning tasks, and activation typically increases as performance improves and is positively correlated with behavioral measures of learning (Seger, Peterson, Cincotta, Lopez-Paniagua, & Anderson, 2010; Cincotta & Seger, 2007; Foerde, Knowlton, & Poldrack, 2006; Seger & Cincotta, 2005, 2006). Moreover, electrophysiological studies indicate that the putamen is crucial for mediating motor responses, especially during late stages of learning (Williams & Eskandar, 2006). On the basis of these proposed roles, we predict greater activity for the visual and motor loops during the stimulus–response phase of categorization.
We have proposed that the executive and motivational corticostriatal loops play an important role in processing feedback and using it to modulate category membership representations (Seger, 2008). Feedback and reward processing recruit regions of the executive and motivational loops in a variety of tasks (Delgado, Locke, Stenger, & Fiez, 2003; Delgado, Nystrom, Fissell, Noll, & Fiez, 2000; Elliott, Sahakian, Michael, Paykel, & Dolan, 1998). The executive loop is comprised of the head of the caudate, lateral pFC, and posterior parietal cortex (Lawrence et al., 1998), whereas the motivational loop consists of the ventral striatum (nucleus accumbens and ventral areas of the caudate and putamen), interacting with ventral frontal cortex (both medial and lateral) as well as the hippocampus and amygdala (Groenewegen, Galis-de Graaf, & Smeets, 1999; Lawrence et al., 1998). Striatal components in both the motivational and executive loops have been shown to be sensitive to feedback valence (Seger et al., 2010; Delgado, Miller, Inati, & Phelps, 2005; Seger & Cincotta, 2005), generally with greater activity for positive than negative feedback in the executive loop (but not the motivational loop). Although the motivational loop plays role in feedback processing, its function differs from that of the executive loop in that the motivational loop is broadly activated by all reward-related activity, regardless of the action performed by the learner (O'Doherty et al., 2004). On the basis of these theorized roles, we expect that the executive and motivational loops will be more active during feedback processing and will be modulated by variables such as feedback valence and expectation.
Interactions between Corticostriatal Loops
Although individual corticostriatal loops perform different functions throughout the course of category learning, these loops must interact in order for categorization to be successful. Anatomical studies have identified at least five different mechanisms through which corticostriatal loops can interact. Two of these follow a functional gradient within the striatum from the most ventro-anterio-medial regions (the ventral striatum) out to the most dorso-postero-lateral regions (posterior putamen and tail of the caudate nucleus), with the anterior caudate and anterior putamen falling in the middle (Voorn, Vanderschuren, Groenewegen, Robbins, & Pennartz, 2004; Haber, Fudge, & McFarland, 2000). The first mechanism is the overlap at the borders between corticostriatal projection fields within the dense dendritic projection fields (Haber, 2009; Draganski et al., 2008). The second is a pattern of recurrent and nonrecurrent feed-forward projections that form loops from striatum to the ventral tegmental area (VTA)/substantia nigra pars compacta (SNc) and back to striatum; Haber likens these loops to an “ascending spiral” (Haber et al., 2000). Recent studies have found that neural activity beginning in one striatal region can induce activity in other neurons along this gradient (Kasanetz, Riquelme, Della-Maggiore, O'Donnell, & Murer, 2008), indicating that these anatomical connections subserve functional interactions. The ventro-antero-medial to dorso-postero-lateral gradient provides a potential mechanism for motivational information in the ventral striatum (motivational loop) to set the organism's state, which is then integrated with knowledge about the current behavioral context (visual loop) and cognitive goals (executive loop), and finally results in the selection and execution of appropriate motor behavior (motor loop). If interactions between corticostriatal loops during categorization learning follow this gradient, then directed influence should progress from the motivational loop to the executive and visual loops and, finally, the motor loop.
In addition, there are three anatomical features of the corticostriatal loops that provide potential mechanisms for more disparate cortical and striatal regions to interact. First, a subset of corticostriatal projection neurons (approximately 15%) extend axons transversely down the striatum that make sparse but spatially widely distributed synaptic contacts across a large population of striatal spiny cells (Haber, 2009; Zheng & Wilson, 2002). Second, some spatially distant but functionally related cortical regions project to the same striatal neurons. A classic example of this is the convergence in projections from the motor and somatosensory representations of digits in the primary motor and primary somatosensory cortices (Flaherty & Graybiel, 1995). Finally, loops may interact via return connections from the thalamus that project to other cortical regions in addition to those that entered the loop, thus forming “open” loops (Joel & Weiner, 1994). This latter mechanism is potentially the most powerful for understanding interloop interactions but is also the least well understood because of the difficulties in tracing return connections from the thalamus to cortex. Some of these open loops follow the ventro-antero-medial to dorso-postero-lateral gradient, as described in Haber (2009). However, at least some open loops connecting cortical regions across a greater distance have been identified. Joel and Weiner (1994) argue that the motor loop has an open loop projection to the executive loop via the substantia nigra pars reticulata and the executive loop has an open loop projection to the motor loop via the internal section of the globus pallidus. In addition, the motivational loop can project to the executive loop via the substantia nigra pars reticulata and possibly to the motor loop via the internal section of the globus pallidus. Of particular note here are the projections running in the opposite direction to the ventro-antero-medial to dorso-postero-lateral gradient through which the motor loop can affect the executive loop and the motivational loop can affect the motor loop. One important example is the feed-forward projection from the visual loop to premotor regions (BA 8) and pre-SMA (Passingham, 1993). Ashby and colleagues' COVIS model proposes that it is this interaction between visual and motor loops that underlies most implicit visual categorization learning (Ashby, Alfonso-Reese, Turken, & Waldron, 1998).
The Current Study
Subjects performed a category learning task, in which stimuli were associated with categories on either a probabilistic or random basis. We implemented jittered intervals between the stimulus–response phase of each trial and feedback to be able to separately model the brain activity associated with each. We hypothesized that the motor and visual loops should be more active during the stimulus–response phase because of the proposed role of these loops in motor selection and stimulus processing, respectively. In contrast, we predicted that the motivational and executive loops should be, overall, more active during feedback processing and sensitive to the valence (positive or negative) and utility of the feedback.
In addition, the current study used measures of effective connectivity obtained via Granger causality mapping (GCM) to visualize exerted influence and, thereby, interactions within and between the four corticostriatal loops during category learning. Previous work in our laboratory examining the interaction between loops was limited to just the motor and executive loops and their respective striatal components, the putamen and head of the caudate (Seger et al., 2010). The current study extends this work to examining all corticostriatal loops including the visual loop and, of particular importance, the motivational loop/ventral striatum as well. If interactions strictly follow the anatomical gradients described above, then the patterns of directed influence should run from motivational to executive to visual to motor loops, which is from the ventral striatum, to the caudate and then to the putamen. Alternatively, the patterns of interaction might follow the information processing demands of the behavioral task: visual evaluation of the stimulus, followed by selection and execution of the appropriate categorical motor response, followed by evaluation of feedback, and finally the integration of feedback with behavior that will provide the mental set necessary for the next trial. If so, we would predict the visual loop would lead to activity in the motor loop, followed by the executive loop, and finally the motivational loop. The association of the motivational loop with holding contextual history and overall motivational context is consistent with recent electrophysiological work (Yamada, Matsumoto, & Kimura, 2007; Williams & Eskandar, 2006).
fMRI participants were 13 members of the Colorado State University (Fort Collins, CO) and School of Medicine, University of Colorado Denver (Aurora, CO) communities. All subjects were healthy, right-handed adults (five men and eight women) with an average age of 27.4 years (range, 23–31 years). Subjects were fluent speakers of English and were screened for a history of neurological and psychiatric disorders, as well as contraindications to MRI (i.e., metallic implants and claustrophobia). Functional data from two additional participants were lost because of technical problems, and one participant was excluded from the GCM analysis because of small timing errors that resulted in stimulus onset and TR onset becoming desynchronized.
Categorization Learning Task
Subjects participated in a categorization task similar to the one used by Seger and Cincotta (2005). A “weather prediction” cover story was used, in which subjects were informed that they should learn which stimuli predict rain and which predict sun. On each trial, participants viewed a single arbitrary visual stimulus (Figure 1) and made a button press with their right hand to categorize the stimulus as being predictive of “rain” or left for “sun.” Following each response, participants were given visual feedback as to whether their response was correct or incorrect. Each stimulus was a fractal image (Seattle Fractals Digital Art, Seattle, WA) created using Tearazon v29 fractal drawing software (Stephen Ferguson, Houston, TX). The stimuli were selected on the basis of being distinctive patterns without easily verbalized patterns or features in common across categories.
The relationship between each stimulus and the two possible outcomes varied in probability. In the probabilistic condition, each was paired with one outcome for 80% of the time and the opposite outcome for 20% of the time (two stimuli were 80/20 sun/rain, and two were 80/20 rain/sun). In the random condition (two stimuli), stimuli were paired with each outcome for 50% of the time. Six stimuli were randomly assigned to each condition for each participant to avoid specific item effects.
For analysis, experimental trials were broken into conditions on the basis of stimulus–outcome association (Probabilistic, Random), correctness of response (Correct, Incorrect), and feedback received (Positive, Negative). Probabilistic stimuli were classified both in terms of whether the participant responded correctly (in accordance with the dominant category assignment) and in terms of the feedback received by the participants resulting in four possible category types: Probabilistic–Correct–Positive (Prob-CP), Probabilistic–Incorrect–Negative (Prob-IN), Probabilistic–Correct–Negative (Prob-CN), and Probabilistic–Incorrect–Positive (Prob-IP). Random stimuli were divided into two categories: Random-Positive (subject received positive feedback) and Random-Negative (subject received negative feedback). Because of the extremely low number of trials (average of 2.6 per subject, with some subjects having experienced none of these trials), Probabilistic–Incorrect–Positive trials were excluded from analysis.
fMRI Image Acquisition
Images were obtained on a research-dedicated 3.0 T whole-body MRI scanner (GE Healthcare, Milwaukee, WI) at the Brain Imaging Center at the University of Colorado Denver (Aurora, CO). The scanner was equipped with an eight-channel, high-resolution-phased array head coil using GE's Array Spatial Sensitivity Encoding Technique software. Anatomical images were collected using a T1-weighted SPGR sequence (minimal repetition time (TR), echo time (TE) = 3.95 msec, inversion time (TI) = 950 msec, flip angle (FA) = 10°, field of view = 220 mm, coronal matrix = 256 × 256; 166 1.2-mm slices). The structural images were used to verify proper slice selection and to determine the sites of functional activation (i.e., voxels that were found to be significantly activated during the functional scan were overlaid on the high-resolution structural images). Functional images were reconstructed from 26 axial oblique slices obtained using a T2*-weighted echo-planar imaging–gradient recalled echo sequence (TR = 1500 msec, TE = 30 msec, FA = 64°, field of view = 220 mm, matrix = 64 × 64 matrix; 4.0-mm slices; no interslice gap) to measure the BOLD signal change. In addition, the first five volumes, recorded before longitudinal magnetization reached a steady state, were discarded.
Visual stimuli were presented to subjects using a magnet-compatible projector that projects visual images onto a mirror attached to the RF head coil. A computer running E-Prime 2.0 experiment software (Psychology Software Tools, Inc., Pittsburgh, PA) was used to control stimulus presentation and interface with two magnet compatible response boxes placed one in each hand. Head movement was minimized using small foam pads placed on each side of the head inside the radio frequency (RF) head coil.
Image analysis was performed using BrainVoyager QX V1.10 (Brain Innovation, Maastricht, the Netherlands). Functional data was first subjected to preprocessing, consisting of (1) three dimensional motion correction using trilinear interpolation, (2) slice scan time correction using cubic spline interpolation, (3) temporal data filtering with a high-pass filter of three cycles in the time course, and (4) linear trend removal. Each subject's high-resolution anatomical image was normalized to the Talairach and Tournoux (1998) brain template. The normalization process consisted of two steps: an initial rigid body translation into the AC–PC plane, followed by an elastic deformation into the standard space performed on 12 individual subvolumes. The resulting set of transformations was applied to the subject's functional image volumes to form volume time course representations to be used in subsequent statistical analyses. Finally, the volume time course representations were spatially smoothed using a Gaussian kernel, FWHM of 6.0 mm.
The experiment used a rapid event-related fMRI design. Trials were arranged pseudorandomly to control for any sequential effects, and “null” fixation events were intermixed with experimental trials to provide a measure of baseline activation (Bandettini, 2007; Donaldson, 2004). The trial length varied depending on a number of factors. First, stimuli were presented for a duration of 1750 msec, in which subjects were required to make their response indicating category membership. Following the response, stimuli were removed without masking, and there was a jittered delay that varied in duration between 250 and 3500 msec optimized using Optseq2 (surfer.nmr.mgh.harvard.edu/optseq; Dale, 1999), before subjects received trial-specific feedback presented for 1000 msec. Four jitter values were used: 250 msec (34 trials), 1750 msec (384 trials), 3250 msec (91 trials), and 4750 msec (31 trials). These jitter values summed with the stimulus presentation length and feedback presentation length to equal a total length of time that was an even multiple of the TR so that stimulus onset was always synchronized with the beginning of a TR. After feedback, the intertrial interval was once again jittered within a range between 1500 and 9000 msec, this time with values randomly sampled from a geometric distribution.
Adding variable intervals of time between stimulus presentation/response selection and feedback, as well as between trials was an important part of this study because the unequal spacing between events allowed the hemodynamic response to vary in how fast it returned to baseline. Thus, varying the duration of these two delays was crucial because it enabled the separation of the hemodynamic responses associated with stimulus presentation/response and feedback. Similar methods were used by Aron and colleagues (2004) to separate the hemodynamic response associated with stimulus and the one associated with feedback. We tried to strike a balance between having a long enough jitter between stimulus–response and feedback to allow us to separately model each of these events while keeping the delay short enough that learning was not substantially impaired. Previous category learning research (Maddox & Ing, 2005; Maddox, Ashby, & Bohil, 2003) found that delays as short as 2500 msec impaired, but did not eliminate, striatally mediated category learning and resulted in some subjects shifting to the use of suboptimal strategies. On most of our trials, the stimulus offset to feedback onset interval was 1750 msec; given that subjects typically made their responses 400 msec before the stimulus offset, this implies an average response to feedback interval of around 2150 msec. We also confirmed that subjects were able to learn despite the delays in pilot studies. As described below in Behavioral Results, subjects were able to learn and even reached asymptotic performance during the first block of training. Given the difficulty in isolating just these two events, as well as recent evidence showing that adding an extended delay between response and feedback can negatively impact category learning (Ell, Ing, & Maddox, 2009), the current study did not try to further dissociate stimulus presentation from response selection, which would have required adding a third window of jittered intervals.
For stimulus-locked analysis, we examined the BOLD signal beginning with the presentation of a stimulus (first TR) of each trial. Similarly, feedback-locked analyses were performed on the TR corresponding with the presentation of feedback, which ranged from the onset of the fourth TR to the onset of the seventh TR within a trial. For both stimulus-locked and feedback-locked analyses, 34 trials in which there was minimal jitter between response and feedback (250 msec). These trials were excluded from our analyses, resulting in a total of 34 of 540 trials being excluded. This was done to minimize any potential overlap between stimulus and feedback and, thus, isolate the hemodynamic response associated with each event.
A whole-brain analysis was performed using the general linear model (GLM) implemented in BrainVoyager QX. The time courses for each condition were convolved with a prototypical hemodynamic function. Conditions were then compared by running the GLM using separate subject predictors, which treated subjects as a random effect (Buchel, Holmes, Rees, & Friston, 1998). A total of 10 conditions were explicitly defined as the combination of the five trial types (Prob-CP, Prob-IN, Prob-CN, Ran-N, and Ran-P, as described above) at the two time points (stimulus onset and feedback onset). Conditions were compared against an implicit baseline constructed of all time points other than the selected conditions.
We used two methods to control for multiple comparisons. Unless otherwise mentioned, the method used was the false discovery rate, with a threshold of q < 0.05 (Genovese, Lazar, & Nichols, 2002). In other cases, we corrected for multiple comparisons using the cluster-size thresholding procedure developed by Forman et al. (1995) extended to 3-D maps and implemented in the Brain Voyager Cluster Threshold plug-in (Goebel, Esposito, & Formisano, 2006). We indicate the use of this method below “corrected alpha < .05.” An initial map was formed using an uncorrected p value of <.005. The minimum cluster size (on the basis of an alpha level of .05) was then set by MonteCarlo simulation using 1000 iterations, simulating the stochastic process of image generation. Afterward, spatial correlations between neighboring voxels were calculated, before voxel intensity thresholds were finally calculated and the corrected map was formed.
Given what is known about the striatal components of individual loops, ROIs within the striatum were defined a priori to perform analyses examining changes in activation associated with feedback across scans and for GCM. Striatal ROIs were anatomically defined on the basis of patterns of activation reported in Delgado and colleagues (2005) and Seger and Cincotta (2005) and were drawn to ensure only gray matter within each striatal region was selected and did not extend into any surrounding white matter or gray matter structures such as thalamus or pallidum. Defining ROIs on the basis of anatomy rather than on the basis of the functional scans has the advantage that it avoids the risk of circular reasoning when voxels defined on a functional bases in one analysis are then subjected to additional analyses (Kriegeskorte, Lindquist, Nichols, Poldrack, & Vul, 2010; Kriegeskorte, Simmons, Bellgowan, & Baker, 2009). ROIs were confirmed to meet these conditions for every subject by individually overlaying ROIs onto each subjects' high-resolution, normalized anatomical image. The resulting ROIs were (1) head of the caudate (encompassing 270 voxels centered on x = −10, y = 9, z = 12), (2) body of the caudate (encompassing 516 voxels centered on x = −16, y = −3, z = 20), (3) tail of the caudate (encompassing 372 voxels centered on x = −27, y = −28, z = 2, (4) putamen (encompassing 268 voxels centered on x = −21, y = 3, z = 9), and (5) ventral striatum (encompassing 443 voxels centered on x = −10, y = 7, z = −5). To aid in GCM, a cortical ROI was created for the anterior cingulate gyrus, which is involved in the executive loop and has been thought to continuously interact with the striatum during instrumental learning. (Ullsperger & von Cramon, 2003; Holroyd & Coles, 2002). Unlike primary motor or visual cortices associated with the motor and visual loops, defining the cortical complements of the executive loop proved to be more difficult, given individual differences in anatomy in the pFC. Hence, the anterior cingulate gyrus ROI (encompassing 628 voxels in both hemispheres but centered on x = −1, y = 10, z = 33) was functionally defined. Finally, all ROIs, except the bilateral anterior cingulate ROI, were then translated horizontally across the x axis to create complementary ROIs for the right hemisphere and, again, individually checked to ensure they were located within gray matter and did not include any white matter. The ROI GLM tool of BrainVoyager QX checking for random effects was used to analyze contrasts between conditions separately within each ROI.
GCM was used to explore effective connectivity between the striatum and other brain regions. This study applied Roebroeck, Formisano, and Goebel's (2005) procedure, as implemented within BrainVoyager, for creating causality maps that provide a measure of directed influence. The ROIs described above were used as reference regions (x) for GCM. Target regions were defined as any voxel not included in the reference region (y). Influence measures were then calculated from the reference to target region (FX→Y), target to reference region (FY→X), and total linear dependence between the reference and target regions (FX,Y) by repeatedly pairing the time course maps of each voxel in these regions. Time course data was sampled from all trials in every condition across three runs. The subsequent GCM analysis was performed using preprocessed data, which included spatial smoothing.
Directed influences to and from the reference region were calculated by subtracting the influence of the target to reference region from the influence from the reference to the target region (FX→Y − FY→X) for every voxel to calculate a difference (dGCM). Thus, effective connectivity was described as: dGCM = FX→Y − FY→X (see Roebroeck et al., 2005, for details). A positive difference value indicates FX→Y (reference→volume) influence, whereas negative difference values depict FY→X (volume→reference) influence. Statistical significance thresholds for the effective connectivity maps were computed by first creating individual maps for each ROI for each subject, then comparing activation across maps using a voxelwise t test examining whether activity was significantly different from zero, at a cluster threshold of p < .05.
Accuracy for probabilistic stimuli was defined in terms of subjects' response according to the dominant category assignment of a stimulus rather than in terms of type of feedback received. As illustrated in Figure 2A, a 2 (Condition: Probabilistic and Random) × 3 (Block) repeated measures ANOVA revealed only a main effect of condition (F(1, 12) = 19.75, p < .01) with higher performance associated with probabilistic trials; the lack of a significant interaction and main effect of block implied that learning occurred early for probabilistic stimuli. To further examine learning during the early stages of the task, one-way repeated measures ANOVA was conducted on probabilistic learning trials within the first run, broken down into quartiles (45 trials each). As illustrated in Figure 2B, this analysis revealed a significant main effect of quartile (F(3, 36) = 6.85, p < .01) as accuracy increased across quartiles. A Bonferonni post hoc test on pairwise comparisons between individual quartiles showed a trend toward a significant difference between the first and second quartiles (p = .06) and a significant difference between the first and third quartiles (p < .01). These results support the notion that probabilistic stimuli were successfully learned during the first scan.
Whole-brain GLM Analyses
For each analysis below, a full list of all activated regions can be found in Supplementary Tables 1–4.
Stimulus–Response Related Activity
To identify neural regions associated with accurate categorization, we compared all probabilistic trials in which stimuli were classified correctly, regardless of feedback received, against trials with incorrect responses (Prob-CP + Prob-CN > Prob-IN) in hopes to further isolate any regions necessary solely for correct categorization, as opposed to regions that are recruited for aspects of the task that are common to both correct and incorrect trials. We found greater activity for correctly classified trials bilaterally in ventrolateral pFC and in the left body of caudate and right head of the caudate (corrected alpha < .05).
We first examined activation associated with positive feedback by comparing correctly categorized trials resulting in positive feedback (Prob-CP) with baseline. We found large clusters of activation across the striatum including the bilateral head of the caudate nucleus, the anterior putamen, and the tail of the caudate. Activation was also found throughout the pFC (bilateral DLPFC, superior frontal gyrus, and anterior cingulate) and the parietal cortex, mainly the paracentral lobule. Similar patterns of activation were observed for random trials resulting in positive feedback (Rand-P > Baseline). Next, we examined trials resulting in negative feedback (Prob-CN, Prob-IN, and Rand-N > Baseline). Overall, the pattern of activity was similar to that of positive feedback: Increased activation observed in the left head of the caudate, left putamen, and right insula. In addition, there was modulation of activity throughout the anterior and posterior cingulate gyrus and hippocampus. Because of the probabilistic nature of some of the stimulus–category relationships, reward expectations leading up to feedback were different for the various types of trials; positive feedback was expected in some trials and the expectations were violated (Prob-CN), whereas on other trials either negative feedback expected in some trials, or there was no clear expectation (e.g., Prob-IN and Ran-N). When we compared these trials (Prob-CN > Prob-IN + Rand-N), unexpected negative feedback was associated with greater activation in the body of caudate, putamen, ventromedial pFC, and regions of the visual cortex (corrected alpha < .05).
We then compared probabilistic trials on which subjects could have learned and, thus, had feedback expectations, with random trials (Prob-CN + Prob-IN > Rand-N). The probabilistic trials recruited areas throughout the pFC (such as medial prefrontal, anterior cingulate, and DLPFC) as well as regions in the body of the caudate and putamen (corrected alpha < .05).
Stimulus–Response versus Feedback
Because we separated discrete events within each trail via our design, it allowed us to directly contrast stimulus–response and feedback processing. We collapsed across feedback type and reward contingency for all probabilistic trials and treated both the epoch used for the stimulus-locked analyses and the one used for feedback-locked analyses as a predictors. The resulting GLM (Stimulus–Response > Feedback) showed greater striatal activation in the ventral striatum extending into the most inferior portion of the head of the caudate and the posterior caudate (see Figure 3). In addition, we found greater activity throughout the bilateral hippocampus, bilateral motor cortex, bilateral visual cortexes, left DLPFC, superior parietal lobe, and left insula. These patterns of results are consistent with our predictions that the visual loop (posterior caudate and visual cortex activation) and motor loop (motor cortex activation) should be primarily recruited during the stimulus–response phase. Reversing the contrast (Feedback > Stimulus–response) did not yield a significant pattern of activation. Interestingly, some regions of the motivational and executive loops (part of the ventral striatum and pFC, respectively) were actually more active in the stimulus–response phase than the feedback phase. We note that there are executive functions required during stimulus–response phase, in particular, during early learning when subjects tend to evaluate each stimulus.
Effect of Valence: Interaction between Positive and Negative Feedback and Scan
We examined the effects of feedback valence via whole-brain contrasts across all scans and follow up ROI analyses that allowed us to identify interactions between valence and scan. We found no significant main effects of valence in the whole brain analysis in contrasts collapsing across all conditions (Prob-CP + Rand-P > Prob-CN + Prob-IN + Rand-N) or examining only random trials (Ran-P > Ran-N). When only probabilistic trials (Prob-CP > Prob-IN) were examined, there were no significant effects of valence within any striatal region, although there was greater activation throughout the mediofrontal cortex and posterior cingulate gyrus for Prob-CP trials. Although some previous studies have found a main effect of feedback valence within the striatum (Delgado et al., 2005; Seger & Cincotta, 2005), others have found an interaction between valence and stage of learning (Seger et al., 2010). We looked for such an interaction between valence and scan within three dorsal striatal ROIs: head of the caudate, body of the caudate, and putamen. Within the right and left heads of the caudate feedback valence interacted with scan. As shown in Figure 4A, during probabilistic learning trials, activation associated with positive feedback (Prob-CP) remained steadily elevated across scans, whereas activation associated with negative feedback (Prob-IN + Prob-CN) increased across scans. A different pattern was exhibited on random trials (Figure 4B): Activation associated with positive feedback (Rand-P) was greater than for negative feedback (Rand-N) in the first block but the difference between positive and negative feedback converged across the time course of learning. In contrast, there was no interaction between feedback valence and scan in the body of the caudate and the putamen; activation remained significantly elevated above baseline throughout the duration of the task for both negative and positive feedback.
Granger Causality Analysis
To explore interactions within and between different corticostriatal loops throughout the course of category learning, we measured effective connectivity using GCM, which allowed us to visualize directed influence between striatal and cortical regions. Our measures of effective connectivity were obtained using the anatomically defined ROIs as seed regions. Figure 5 shows the overall patterns of directed influence between striatal regions, and Figure 6 shows patterns of interaction between cortical and striatal regions within each loop. A complete list of areas of directed influence to and from each reference region can be found in Supplementary Table 5.
Head of the Caudate
As illustrated in Figure 6, second column, the head of the caudate exerted directed influence on the parietal and pFC and received directed influence from several regions along the anterior cingulate gyrus. Within the striatum, the head of the caudate received directed influence from the putamen and exerted directed influence on the body and tail of the caudate, as shown in Figure 5, third column.
Along with the head of the caudate, we measured effective connectivity to and from a region in the anterior cingulate for a more comprehensive look at patterns of activity within the executive loop. The small seed region within the anterior cingulate exerted directed influence on both head and body/tail regions of the caudate nucleus. In addition, directed influence was observed from prefrontal areas, ventral tegmental area, as well as small clusters of activation along the cingulate gyrus.
Body and Tail of the Caudate
We examined two pairs of ROIs in the posterior caudate, one in the body and one in the tail region. The tail of the caudate ROIs received directed influence from higher-order visual areas in the occipital lobe as well as the fusiform gyrus in the inferior temporal lobe (Figure 6, column 4, rows 1, 2, and 3). In addition, the body of the caudate received directed influence from the anterior cingulate and several other cortical areas such as DLPFC and inferior frontal gyrus (Figure 5, column 3), and the tail of the caudate received directed influence from the motor cortex. Both body and tail regions exerted directed influence on further posterior regions of the posterior caudate and received directed influence from the head of the caudate and putamen. In addition, the tail but not the body of the caudate exerted directed influence on the posterior hippocampus and received directed influence from the thalamus.
From our reference regions in the anterior putamen, directed influence was observed in the primary motor cortex (Figure 6, column 1), consistent with the known anatomy of the motor loop, as well as several cortical regions in the parietal and frontal lobes. Within the BG, the putamen exerted directed influence bilaterally to the body of the caudate and pallidum, as seen in Figure 5. We did not find any directed influence into the reference region.
The ventral striatum exerted directed influence on a region of ventromedial frontal cortex (Figure 5, column 4) and received directed influence from the bilateral OFC and hippocampus (Figure 6, column 4, row 4); these patterns of influence are consistent with the known anatomy of the motivational loop. Interestingly, the ventral striatum also exerted directed influence on a very small region of the inferior temporal gyrus, part of the visual loop. Within the BG, the ventral striatum received directed influence from several regions of dorsal striatum, including the head of the caudate and ventral putamen.
By dissociating stimulus–response from feedback processing, this study was able to help clarify the shared and individual roles played by corticostriatal loops during visual categorization learning. We found that all the corticostriatal loops were recruited during both stimulus–response and feedback processing, implying that each has a role in both phases of categorization. However, the visual and motor loops were more associated with stimulus–response than feedback processing (as shown in the direct contrast of Stimulus–Response > Feedback), whereas the executive loop was sensitive to feedback processing (as shown in the interaction between feedback valence and scan). Furthermore, GCM revealed both interactions between cortical and striatal regions within individual corticostriatal loops and distinct patterns of influence between loops; in particular, the motor loop exerted influence on the executive and visual loops, and all loops exerted directed influence on the motivational loop.
Striatal regions within the visual loop, the body and tail of the caudate, were active during both stimulus–response and feedback processing. However, there was greater activation in the tail of the caudate nucleus during the stimulus–response epoch, which is consistent with higher visual categorization demands in this phase of the task. The body was more active for correct than incorrect trials, consistent with it being associated with successful learning (Nomura et al., 2007; Foerde et al., 2006; Seger & Cincotta, 2005, 2006). These findings are consistent with the theory that the posterior caudate, along with regions in the middle and inferior temporal lobe, are necessary for processing complex visual stimuli, and recruitment of this region is important for successful category learning (Seger et al., 2010; Seger, 2008).
GCM showed directed influence between regions within the visual loop. The tail of the caudate received directed influence from higher-order visual areas in both occipital and inferior parietal lobes. Within the visual loop, both seed regions exerted directed influence relatively posterior regions of the caudate. In addition, the body of the caudate received influence from cortical regions involved in the executive and motor loops, including the anterior cingulate gyrus, and prefrontal cortices. This indicates a possible intermediate functional role for the body of the caudate between the executive and visual loops, which is consistent with its anatomical position between the head and tail of the caudate.
Within the motor loop, the putamen and output nuclei of the BG, such as the pallidum, were active across all conditions in both stimulus–response and feedback phases. Cortical regions involved in the motor loop were more active during stimulus–response than feedback, which is consistent with the greater role of the motor loop in this phase of categorization learning. Interestingly, we did not find an overall difference in putamen activation between phases of categorization learning. The putamen, however, was recruited to different degrees during feedback processing contrasts that differed in whether subjects knew the answer or not; it was more active when subjects were correct (Prob-CN) than when they were incorrect or could not know the answer (Prob-IN and Ran-N) and more active overall when the stimulus–category relationships were learnable (Prob-CN and Prob-IN) than when they were random (Ran-N). These patterns are consistent with the putamen being associated with well-learned associations (Ashby, Turner, & Horvitz, 2010; Daniel & Pollmann, 2010; Seger et al., 2010).
The putamen exerted influence on cortical regions within the motor loop including the motor cortex. During learning, the motor loop is thought to facilitate the selection and execution of appropriate motor responses depending on the task (Seger et al., 2010; Seger, 2008). In addition, the putamen exerted directed influence on regions of the caudate and cortex associated with the executive loop, consistent with an overall influence of the motor loop on the executive loop.
Within the executive loop, we found recruitment of the anterior caudate nucleus in both stimulus–response and feedback phases of categorization and across all trial types. GCM revealed interactions between striatal and cortical regions within the executive loop. The anterior caudate nucleus exerted directed influence on lateral pFC and superior parietal lobule while receiving influence from the anterior cingulate gyrus.
The response to negative and positive feedback changed across the time course of learning. For probabilistic trials, activity levels for positive and negative feedback were similar in the first scan, but activity associated with positive feedback decreased and activation associated with negative feedback sharply increased across scans. In contrast, for random trials, there was greater activity for positive than negative feedback on Scan 1 that disappeared in subsequent scans. These results indicate that feedback processing in the head of the head of the caudate is not simply a function of feedback. Instead, activity patterns follow the utility of the feedback. In early stages of learning and/or when feedback is random, there is often an overall greater activity for positive feedback, as we found in Scan 1 for the random condition (Delgado et al., 2005; Seger & Cincotta, 2005). However, in situations when negative feedback is less expected and more useful, there is greater activity for negative than positive. The increase in activation associated with negative feedback in the late stages of learning is in line with other tasks involving probabilistic learning or task-switching (Bischoff-Grethe, Hazeltine, Bergren, Ivry, & Grafton, 2009; Monchi et al., 2001), in which negative feedback can become useful signal to indicate a change in rule acquisition strategy.
The pattern of activity we find in the head of the caudate are compatible with studies specifically looking at how the dorsal striatum is involved in acquiring and updating stimulus–response associations that lead to reward (Seger et al., 2010; O'Doherty et al., 2004; Tricomi, Delgado, & Fiez, 2004) via interactions with cortical reward centers (Kringelbach, 2004; Knutson, Fong, Bennett, Adams, & Hommer, 2003). In studies incorporating reinforcement learning modeling, activity in the anterior caudate and putamen is sensitive to “reward prediction” or the expected feedback associated with making the categorical response, whereas the ventral striatum reflects “prediction error” or the difference between expected feedback and actual feedback (Seger et al., 2010).
In addition to the recruitment of the anterior caudate, there were also interesting patterns of activation in cortical regions that interact with this part of the striatum in the executive loop. The ventrolateral pFC was more active for correct than incorrect responses, consistent with theories proposing a role for these region in visual working memory (Goldman-Rakic, 1987) and in mediating between sensory areas and motor areas (Miller, 2000; Fuster, 1997). The anterior cingulate and DLPFC showed greater activity during stimulus–response processing than during feedback.
The ventral striatum was significantly more active during stimulus–response than the feedback phase of learning. This pattern runs counter to our initial predictions, which were that the motivational loop should play a primary role at the time of feedback processing. However, this finding complements the results of the GCM, in which we found that the motivational loop received directed influence from all the other corticostriatal loops. These two findings imply that the motivational loop plays a role in integrating all the information about a trial and using it to provide the motivational “set” for the following trial. In this sense, the motivational loop can be seen as a connection between trials, maintaining the history of past reinforcement to be used in the upcoming trials. Previous electrophysiological research in monkeys has found patterns of activity in the ventral striatum consistent with this role in maintaining motivational history or set across trials (Wickens, Budd, Hyland, & Arbuthnott, 2007; Yamada et al., 2007), and researchers have argued that the ventral striatum plays a role in selecting appropriate actions (Nicola, 2007). Interestingly, we found that the ventral striatum exerts a directed influence on visual cortex, which is also consistent with the ventral striatum affecting the functions of the visual loop during stimulus–response processing.
Expectation and Prediction Error in the Striatum
An influential theory that accounts for many (but not all) aspects of striatal function comes from the field of reinforcement learning. Reinforcement learning hypothesizes that agents make predictions about the value of different options (expected reward or reward prediction) and use these values to decide which option to take. After receiving feedback, the agent calculates the difference between the expected and obtained reward, referred to as reward prediction error. Prediction error can then serve as a teaching signal to modify value representations to reflect experience. Reinforcement learning provides a good account of the firing patterns of at least some subpopulations of dopamine neurons from the midbrain (VTA and SN). These neurons project to the whole of the striatum, with densest innervation of the ventro-anteror-medial portions of the striatum, and falling off in intensity across much the same gradient as the anatomical corticostriatal loops. Studies of the BG find that activity in the ventral striatum is well characterized by prediction error, whereas activity in the dorsal striatum, particularly the putamen, is accounted for by reward prediction. The intermediate portions of the corticostriatal loops (head of the caudate/executive loop) have a mixed pattern in which it can be sensitive to both aspects of learning (Haruno & Kawato, 2006; O'Doherty, 2004).
Overall, activity in the head of the caudate nucleus is broadly consistent with the core idea that the striatum represents reward and is particularly sensitive to violations of reward prediction. However, the activation patterns we found are not consistent with a strict model of reinforcement learning and require additional processes to account for them. In this study, subjects learned quickly, and as a result, reward prediction values and prediction error values (when unexpected feedback was obtained) were constant across Blocks 1–3. When negative feedback was received, it was approximately equally likely that it was expected (correctly indicating a mistake) as that it was unexpected. Therefore, if our results were driven primarily by prediction error, we should have found no differences in activity resulting from feedback across blocks. However, as shown in Figure 4, we actually found substantial differences. First, we found for the probabilistic stimuli a striking increase in activity associated with negative feedback on the final block that was not present in Blocks 1 and 2. Reinforcement learning would predict that there should be consistently higher activity for negative feedback over positive across all three blocks. Second, for random stimuli, we found initially higher activity for positive feedback in Block 1 that disappeared in following blocks. Because random stimuli are unlearnable, unexpected positive feedback should be just as likely as unexpected negative feedback, and there should be no valence-based difference and no change across blocks. To account for these differences, it will be necessary to modify traditional reinforcement learning models to include states that can account for cognitive knowledge that goes beyond knowledge of the reward associated with the possible category membership responses for each stimulus. Extensions to reinforcement learning are an active area of research (Zhang, Berridge, Tindell, Smith, & Aldridge, 2009). One promising approach is to add a representation of cognitive state (which can reflect consciously held beliefs or biases) to reinforcement learning models (Glascher, Daw, Dayan, & O'Doherty, 2010).
Interactions between Corticostriatal Loops
Using GCM, we observed that the putamen exerted directed influence on the anterior dorsal caudate nucleus, which in turn exerted directed influence on further posterior regions of the caudate. In addition, both the putamen and dorsal caudate exerted directed influence on the ventral striatum. Thus, the overall pattern we found via the striatal seed regions was putamen > anterior caudate > posterior caudate > ventral striatum or, in terms of loops, motor > executive > visual > motivational. We also found some patterns of directed influence with cortex that were consistent with this pattern. In particular, there was directed influence from visual cortex to the ventral striatum, directed influence from motor cortex to the tail of the caudate, and directed influence from anterior cingulate to the head and body–tail regions of the caudate nucleus.
Overall, the patterns of interaction between loops we observed best fits our functional predictions on the basis of the sequence of events involved in category learning: categorical response (motor) followed by feedback processing (executive) and the updating of reward predictions in preparation for the next trial (motivational). The intermediate position of the executive loop implies that it interacts with both the motivational and motor loops to facilitate the integration of stimulus–response–outcome associations. These interactions are consistent with the open loop projections between these loops described by Joel and Weiner (1994).
Our results are inconsistent with the anatomical gradient from motivational to executive/visual to motor loops (Haber et al., 2000) in two ways. First, the anatomical gradient would imply that the executive loop should affect the motor loop rather than the opposite pattern we found, which was additionally consistent with previous work in our laboratory (Seger et al., 2010). Second, the motivational loop was influenced by all the other loops rather than being the source of all influence: It was at the top of the hierarchy rather than the bottom. However, the motivation loop likely performs both roles: It is the base of the hierarchy in the sense that it holds the motivational context within which the upcoming stimulus is processed and responded to, and in addition, it is at the top of the hierarchy in the sense that it integrates all the information from the trial to serve as the basis for future trials. This dual role is consistent with electrophysiological research, indicating that the ventral striatum can act based on history of past reinforcement to update reward prediction (Wickens et al., 2007; Yamada et al., 2007).
One limitation of GCM in this study is that Granger causality was performed over all trials across three scans; therefore, it is not possible to separate and compare connectivity associated with specific conditions. In addition, the relationships between neural regions revealed by GCM are subject to influence by separate regions not accounted for in the model. Because directed influence is a measure that uses time course values in the seed region to predict values in a target region (and vice versa), directed influence does not take into account the possibility that influence from one region to another may not be direct but rather be mediated by other regions. In addition, another limitation of GCM is that it assumes a homogeneous hemodynamic response in all regions. Because the BOLD response can vary from region to region within the same subject, results from GCM should be interpreted with caution (David et al., 2008). Finally, it should be noted that our GCM analyses do not correct for multiple comparisons and should, thus, be interpreted with caution.
Our results illustrate the dynamic interactions between corticostriatal loops during learning. All loops were recruited during both stimulus–response and feedback processing, emphasizing the need to integrate all three aspects during learning. However, there were some differences in recruitment: The visual and motivational loops were more active during stimulus–response processing than feedback processing, indicating a greater role in determining category membership. The executive loop was sensitive to feedback valence, indicating a greater role in feedback processing. Directed influence progressed from the motor loop to the executive loop, consistent with integration of categorical motor response with its outcome. In addition, both motor and executive loops influenced the motivational loop, providing further evidence that the motivational loop stores history of reward across trials.
This research was supported by a grant from the National Institutes of Health (R01MH079182). We thank Erik Peterson, Lucy Troup, and Bruce Draper for thoughtful discussion.
Reprint requests should be sent to Carol A. Seger, Department of Psychology, 1876 Campus Delivery, Colorado State University, Fort Collins, CO 80523, or via e-mail: Carol.Seger@colostate.edu.