Abstract

We examined dynamic interactions between cortex and BG during stimulus–response and feedback processing phases of categorization. First, we dissociated stimulus–response processing from feedback processing using “jittered” intervals of time between response and feedback to examine how each recruits the four primary corticostriatal loops (motor, executive, visual, and motivational). Second, we examined dynamic interactions within and between corticostriatal loops using Granger causality mapping. On each trial, subjects viewed one of six abstract visual stimuli, pressed a button indicating category membership, and then received feedback as to whether the decision was right or wrong. Stimulus–response processing was associated with greater activity in the visual loop, whereas feedback processing resulted in activity in the executive loop that was sensitive to feedback valence. Granger causality mapping showed patterns of directed influence within corticostriatal loops and between loops from the motor to the executive, to the visual, and finally to the motivational loop. These patterns of interaction are consistent with functional integration of motor processing in the motor loop with feedback processing in the executive loop and maintenance of stimulus–response history for future responses in the motivational loop.

INTRODUCTION

Categorization is the process of learning to distinguish between different types or groups of objects and situations in the world. Previous studies have identified the BG as critical for category learning in both human and nonhuman animals (Seger & Miller, 2010; Shohamy, Myers, Kalanithi, & Gluck, 2008; Poldrack et al., 2001; Poldrack, Prabhakaran, Seger, & Gabrieli, 1999; Knowlton, Mangels, & Squire, 1996). Along with the BG, various cortical areas have been implicated in category learning (Fellows & Farah, 2003; Cools, Clark, Owen, & Robbins, 2002; Monchi, Petrides, Petre, Worsley, & Dagher, 2001; Lombardi et al., 1999). Our focus here is on examining the functions of corticostriatal systems, with the goal of understanding the division of labor across individual cortical and BG regions and how these regions interact to support learning.

Stimulus–Response and Feedback Processing in the Corticostriatal Loops

Typical trial and error visual categorization learning tasks can be divided into two phases: the stimulus–response phase and the feedback processing phase. In the former, the subject views a stimulus, decides on its category membership, and executes the appropriate motor response indicating the category membership of the stimulus. In the latter, subjects receive feedback indicating whether their response was correct or incorrect and use this information to update their knowledge of the correct category membership. The first goal of the present study is to dissociate neural activity associated with stimulus processing and response selection from that associated with feedback processing within the same task by manipulating the temporal delay between these phases and extend previous work by directly comparing these separate events. This approach was similar to that used by Aron and colleagues (2004) as well as Daniel and Pollmann (2010); however, neither study reported direct comparisons between stimulus–response and feedback phases. This study extends this previous work by directly comparing these separate events, with an emphasis on striatal activity. More importantly, we use this approach of separating stimulus and feedback events to better examine how the cortex and striatum work in parallel to subserve category learning.

Anatomically, the striatum interacts with cortex and other BG regions via recurrent networks referred to as “corticostriatal loops.” Cortical regions project to the striatum, which in turn project to the output nuclei of the BG, and then to the thalamus before returning to cortex. We divide the corticostriatal system into four loops: motivational, executive, visual, and motor (Seger, 2008; Lawrence, Sahakian, & Robbins, 1998). However, it should be noted that these loops overlap and there are no discrete boundaries between loops (Alexander, DeLong, & Strick, 1986).

During category learning, we have proposed that the visual loop helps select the appropriate visual representations of stimuli and map them to appropriate behaviors (Seger, 2008). The visual loop is comprised of cortical regions that underlie visual processing such as the extrastriate occipital lobe, inferior temporal cortex, middle temporal cortex, and FEFs. These cortical regions project to the posterior part of the caudate nucleus. The motor loop is thought to facilitate the selection and execution of appropriate motor responses depending on the task demands (Seger, 2008). The motor loop consists of the putamen interacting with primary motor cortex, somatosensory cortex, and motor planning areas such as the premotor and supplementary motor cortices. Both the motor and visual corticostriatal loops are active during categorization learning tasks, and activation typically increases as performance improves and is positively correlated with behavioral measures of learning (Seger, Peterson, Cincotta, Lopez-Paniagua, & Anderson, 2010; Cincotta & Seger, 2007; Foerde, Knowlton, & Poldrack, 2006; Seger & Cincotta, 2005, 2006). Moreover, electrophysiological studies indicate that the putamen is crucial for mediating motor responses, especially during late stages of learning (Williams & Eskandar, 2006). On the basis of these proposed roles, we predict greater activity for the visual and motor loops during the stimulus–response phase of categorization.

We have proposed that the executive and motivational corticostriatal loops play an important role in processing feedback and using it to modulate category membership representations (Seger, 2008). Feedback and reward processing recruit regions of the executive and motivational loops in a variety of tasks (Delgado, Locke, Stenger, & Fiez, 2003; Delgado, Nystrom, Fissell, Noll, & Fiez, 2000; Elliott, Sahakian, Michael, Paykel, & Dolan, 1998). The executive loop is comprised of the head of the caudate, lateral pFC, and posterior parietal cortex (Lawrence et al., 1998), whereas the motivational loop consists of the ventral striatum (nucleus accumbens and ventral areas of the caudate and putamen), interacting with ventral frontal cortex (both medial and lateral) as well as the hippocampus and amygdala (Groenewegen, Galis-de Graaf, & Smeets, 1999; Lawrence et al., 1998). Striatal components in both the motivational and executive loops have been shown to be sensitive to feedback valence (Seger et al., 2010; Delgado, Miller, Inati, & Phelps, 2005; Seger & Cincotta, 2005), generally with greater activity for positive than negative feedback in the executive loop (but not the motivational loop). Although the motivational loop plays role in feedback processing, its function differs from that of the executive loop in that the motivational loop is broadly activated by all reward-related activity, regardless of the action performed by the learner (O'Doherty et al., 2004). On the basis of these theorized roles, we expect that the executive and motivational loops will be more active during feedback processing and will be modulated by variables such as feedback valence and expectation.

Interactions between Corticostriatal Loops

Although individual corticostriatal loops perform different functions throughout the course of category learning, these loops must interact in order for categorization to be successful. Anatomical studies have identified at least five different mechanisms through which corticostriatal loops can interact. Two of these follow a functional gradient within the striatum from the most ventro-anterio-medial regions (the ventral striatum) out to the most dorso-postero-lateral regions (posterior putamen and tail of the caudate nucleus), with the anterior caudate and anterior putamen falling in the middle (Voorn, Vanderschuren, Groenewegen, Robbins, & Pennartz, 2004; Haber, Fudge, & McFarland, 2000). The first mechanism is the overlap at the borders between corticostriatal projection fields within the dense dendritic projection fields (Haber, 2009; Draganski et al., 2008). The second is a pattern of recurrent and nonrecurrent feed-forward projections that form loops from striatum to the ventral tegmental area (VTA)/substantia nigra pars compacta (SNc) and back to striatum; Haber likens these loops to an “ascending spiral” (Haber et al., 2000). Recent studies have found that neural activity beginning in one striatal region can induce activity in other neurons along this gradient (Kasanetz, Riquelme, Della-Maggiore, O'Donnell, & Murer, 2008), indicating that these anatomical connections subserve functional interactions. The ventro-antero-medial to dorso-postero-lateral gradient provides a potential mechanism for motivational information in the ventral striatum (motivational loop) to set the organism's state, which is then integrated with knowledge about the current behavioral context (visual loop) and cognitive goals (executive loop), and finally results in the selection and execution of appropriate motor behavior (motor loop). If interactions between corticostriatal loops during categorization learning follow this gradient, then directed influence should progress from the motivational loop to the executive and visual loops and, finally, the motor loop.

In addition, there are three anatomical features of the corticostriatal loops that provide potential mechanisms for more disparate cortical and striatal regions to interact. First, a subset of corticostriatal projection neurons (approximately 15%) extend axons transversely down the striatum that make sparse but spatially widely distributed synaptic contacts across a large population of striatal spiny cells (Haber, 2009; Zheng & Wilson, 2002). Second, some spatially distant but functionally related cortical regions project to the same striatal neurons. A classic example of this is the convergence in projections from the motor and somatosensory representations of digits in the primary motor and primary somatosensory cortices (Flaherty & Graybiel, 1995). Finally, loops may interact via return connections from the thalamus that project to other cortical regions in addition to those that entered the loop, thus forming “open” loops (Joel & Weiner, 1994). This latter mechanism is potentially the most powerful for understanding interloop interactions but is also the least well understood because of the difficulties in tracing return connections from the thalamus to cortex. Some of these open loops follow the ventro-antero-medial to dorso-postero-lateral gradient, as described in Haber (2009). However, at least some open loops connecting cortical regions across a greater distance have been identified. Joel and Weiner (1994) argue that the motor loop has an open loop projection to the executive loop via the substantia nigra pars reticulata and the executive loop has an open loop projection to the motor loop via the internal section of the globus pallidus. In addition, the motivational loop can project to the executive loop via the substantia nigra pars reticulata and possibly to the motor loop via the internal section of the globus pallidus. Of particular note here are the projections running in the opposite direction to the ventro-antero-medial to dorso-postero-lateral gradient through which the motor loop can affect the executive loop and the motivational loop can affect the motor loop. One important example is the feed-forward projection from the visual loop to premotor regions (BA 8) and pre-SMA (Passingham, 1993). Ashby and colleagues' COVIS model proposes that it is this interaction between visual and motor loops that underlies most implicit visual categorization learning (Ashby, Alfonso-Reese, Turken, & Waldron, 1998).

The Current Study

Subjects performed a category learning task, in which stimuli were associated with categories on either a probabilistic or random basis. We implemented jittered intervals between the stimulus–response phase of each trial and feedback to be able to separately model the brain activity associated with each. We hypothesized that the motor and visual loops should be more active during the stimulus–response phase because of the proposed role of these loops in motor selection and stimulus processing, respectively. In contrast, we predicted that the motivational and executive loops should be, overall, more active during feedback processing and sensitive to the valence (positive or negative) and utility of the feedback.

In addition, the current study used measures of effective connectivity obtained via Granger causality mapping (GCM) to visualize exerted influence and, thereby, interactions within and between the four corticostriatal loops during category learning. Previous work in our laboratory examining the interaction between loops was limited to just the motor and executive loops and their respective striatal components, the putamen and head of the caudate (Seger et al., 2010). The current study extends this work to examining all corticostriatal loops including the visual loop and, of particular importance, the motivational loop/ventral striatum as well. If interactions strictly follow the anatomical gradients described above, then the patterns of directed influence should run from motivational to executive to visual to motor loops, which is from the ventral striatum, to the caudate and then to the putamen. Alternatively, the patterns of interaction might follow the information processing demands of the behavioral task: visual evaluation of the stimulus, followed by selection and execution of the appropriate categorical motor response, followed by evaluation of feedback, and finally the integration of feedback with behavior that will provide the mental set necessary for the next trial. If so, we would predict the visual loop would lead to activity in the motor loop, followed by the executive loop, and finally the motivational loop. The association of the motivational loop with holding contextual history and overall motivational context is consistent with recent electrophysiological work (Yamada, Matsumoto, & Kimura, 2007; Williams & Eskandar, 2006).

METHODS

Participants

fMRI participants were 13 members of the Colorado State University (Fort Collins, CO) and School of Medicine, University of Colorado Denver (Aurora, CO) communities. All subjects were healthy, right-handed adults (five men and eight women) with an average age of 27.4 years (range, 23–31 years). Subjects were fluent speakers of English and were screened for a history of neurological and psychiatric disorders, as well as contraindications to MRI (i.e., metallic implants and claustrophobia). Functional data from two additional participants were lost because of technical problems, and one participant was excluded from the GCM analysis because of small timing errors that resulted in stimulus onset and TR onset becoming desynchronized.

Categorization Learning Task

Subjects participated in a categorization task similar to the one used by Seger and Cincotta (2005). A “weather prediction” cover story was used, in which subjects were informed that they should learn which stimuli predict rain and which predict sun. On each trial, participants viewed a single arbitrary visual stimulus (Figure 1) and made a button press with their right hand to categorize the stimulus as being predictive of “rain” or left for “sun.” Following each response, participants were given visual feedback as to whether their response was correct or incorrect. Each stimulus was a fractal image (Seattle Fractals Digital Art, Seattle, WA) created using Tearazon v29 fractal drawing software (Stephen Ferguson, Houston, TX). The stimuli were selected on the basis of being distinctive patterns without easily verbalized patterns or features in common across categories.

Figure 1. 

Stimuli used in the experiment. Stimuli were paired with one of two outcomes (“rain” or “sun”) on either a probabilistic reward schedule (four stimuli) in which one outcome was associated with positive feedback 80% of the time or a random reward schedule (two stimuli). Stimuli were randomly assigned to either condition for each participant.

Figure 1. 

Stimuli used in the experiment. Stimuli were paired with one of two outcomes (“rain” or “sun”) on either a probabilistic reward schedule (four stimuli) in which one outcome was associated with positive feedback 80% of the time or a random reward schedule (two stimuli). Stimuli were randomly assigned to either condition for each participant.

The relationship between each stimulus and the two possible outcomes varied in probability. In the probabilistic condition, each was paired with one outcome for 80% of the time and the opposite outcome for 20% of the time (two stimuli were 80/20 sun/rain, and two were 80/20 rain/sun). In the random condition (two stimuli), stimuli were paired with each outcome for 50% of the time. Six stimuli were randomly assigned to each condition for each participant to avoid specific item effects.

For analysis, experimental trials were broken into conditions on the basis of stimulus–outcome association (Probabilistic, Random), correctness of response (Correct, Incorrect), and feedback received (Positive, Negative). Probabilistic stimuli were classified both in terms of whether the participant responded correctly (in accordance with the dominant category assignment) and in terms of the feedback received by the participants resulting in four possible category types: Probabilistic–Correct–Positive (Prob-CP), Probabilistic–Incorrect–Negative (Prob-IN), Probabilistic–Correct–Negative (Prob-CN), and Probabilistic–Incorrect–Positive (Prob-IP). Random stimuli were divided into two categories: Random-Positive (subject received positive feedback) and Random-Negative (subject received negative feedback). Because of the extremely low number of trials (average of 2.6 per subject, with some subjects having experienced none of these trials), Probabilistic–Incorrect–Positive trials were excluded from analysis.

fMRI Image Acquisition

Images were obtained on a research-dedicated 3.0 T whole-body MRI scanner (GE Healthcare, Milwaukee, WI) at the Brain Imaging Center at the University of Colorado Denver (Aurora, CO). The scanner was equipped with an eight-channel, high-resolution-phased array head coil using GE's Array Spatial Sensitivity Encoding Technique software. Anatomical images were collected using a T1-weighted SPGR sequence (minimal repetition time (TR), echo time (TE) = 3.95 msec, inversion time (TI) = 950 msec, flip angle (FA) = 10°, field of view = 220 mm, coronal matrix = 256 × 256; 166 1.2-mm slices). The structural images were used to verify proper slice selection and to determine the sites of functional activation (i.e., voxels that were found to be significantly activated during the functional scan were overlaid on the high-resolution structural images). Functional images were reconstructed from 26 axial oblique slices obtained using a T2*-weighted echo-planar imaging–gradient recalled echo sequence (TR = 1500 msec, TE = 30 msec, FA = 64°, field of view = 220 mm, matrix = 64 × 64 matrix; 4.0-mm slices; no interslice gap) to measure the BOLD signal change. In addition, the first five volumes, recorded before longitudinal magnetization reached a steady state, were discarded.

Visual stimuli were presented to subjects using a magnet-compatible projector that projects visual images onto a mirror attached to the RF head coil. A computer running E-Prime 2.0 experiment software (Psychology Software Tools, Inc., Pittsburgh, PA) was used to control stimulus presentation and interface with two magnet compatible response boxes placed one in each hand. Head movement was minimized using small foam pads placed on each side of the head inside the radio frequency (RF) head coil.

Image Processing

Image analysis was performed using BrainVoyager QX V1.10 (Brain Innovation, Maastricht, the Netherlands). Functional data was first subjected to preprocessing, consisting of (1) three dimensional motion correction using trilinear interpolation, (2) slice scan time correction using cubic spline interpolation, (3) temporal data filtering with a high-pass filter of three cycles in the time course, and (4) linear trend removal. Each subject's high-resolution anatomical image was normalized to the Talairach and Tournoux (1998) brain template. The normalization process consisted of two steps: an initial rigid body translation into the AC–PC plane, followed by an elastic deformation into the standard space performed on 12 individual subvolumes. The resulting set of transformations was applied to the subject's functional image volumes to form volume time course representations to be used in subsequent statistical analyses. Finally, the volume time course representations were spatially smoothed using a Gaussian kernel, FWHM of 6.0 mm.

Design

The experiment used a rapid event-related fMRI design. Trials were arranged pseudorandomly to control for any sequential effects, and “null” fixation events were intermixed with experimental trials to provide a measure of baseline activation (Bandettini, 2007; Donaldson, 2004). The trial length varied depending on a number of factors. First, stimuli were presented for a duration of 1750 msec, in which subjects were required to make their response indicating category membership. Following the response, stimuli were removed without masking, and there was a jittered delay that varied in duration between 250 and 3500 msec optimized using Optseq2 (surfer.nmr.mgh.harvard.edu/optseq; Dale, 1999), before subjects received trial-specific feedback presented for 1000 msec. Four jitter values were used: 250 msec (34 trials), 1750 msec (384 trials), 3250 msec (91 trials), and 4750 msec (31 trials). These jitter values summed with the stimulus presentation length and feedback presentation length to equal a total length of time that was an even multiple of the TR so that stimulus onset was always synchronized with the beginning of a TR. After feedback, the intertrial interval was once again jittered within a range between 1500 and 9000 msec, this time with values randomly sampled from a geometric distribution.

Adding variable intervals of time between stimulus presentation/response selection and feedback, as well as between trials was an important part of this study because the unequal spacing between events allowed the hemodynamic response to vary in how fast it returned to baseline. Thus, varying the duration of these two delays was crucial because it enabled the separation of the hemodynamic responses associated with stimulus presentation/response and feedback. Similar methods were used by Aron and colleagues (2004) to separate the hemodynamic response associated with stimulus and the one associated with feedback. We tried to strike a balance between having a long enough jitter between stimulus–response and feedback to allow us to separately model each of these events while keeping the delay short enough that learning was not substantially impaired. Previous category learning research (Maddox & Ing, 2005; Maddox, Ashby, & Bohil, 2003) found that delays as short as 2500 msec impaired, but did not eliminate, striatally mediated category learning and resulted in some subjects shifting to the use of suboptimal strategies. On most of our trials, the stimulus offset to feedback onset interval was 1750 msec; given that subjects typically made their responses 400 msec before the stimulus offset, this implies an average response to feedback interval of around 2150 msec. We also confirmed that subjects were able to learn despite the delays in pilot studies. As described below in Behavioral Results, subjects were able to learn and even reached asymptotic performance during the first block of training. Given the difficulty in isolating just these two events, as well as recent evidence showing that adding an extended delay between response and feedback can negatively impact category learning (Ell, Ing, & Maddox, 2009), the current study did not try to further dissociate stimulus presentation from response selection, which would have required adding a third window of jittered intervals.

For stimulus-locked analysis, we examined the BOLD signal beginning with the presentation of a stimulus (first TR) of each trial. Similarly, feedback-locked analyses were performed on the TR corresponding with the presentation of feedback, which ranged from the onset of the fourth TR to the onset of the seventh TR within a trial. For both stimulus-locked and feedback-locked analyses, 34 trials in which there was minimal jitter between response and feedback (250 msec). These trials were excluded from our analyses, resulting in a total of 34 of 540 trials being excluded. This was done to minimize any potential overlap between stimulus and feedback and, thus, isolate the hemodynamic response associated with each event.

Whole-brain Analysis

A whole-brain analysis was performed using the general linear model (GLM) implemented in BrainVoyager QX. The time courses for each condition were convolved with a prototypical hemodynamic function. Conditions were then compared by running the GLM using separate subject predictors, which treated subjects as a random effect (Buchel, Holmes, Rees, & Friston, 1998). A total of 10 conditions were explicitly defined as the combination of the five trial types (Prob-CP, Prob-IN, Prob-CN, Ran-N, and Ran-P, as described above) at the two time points (stimulus onset and feedback onset). Conditions were compared against an implicit baseline constructed of all time points other than the selected conditions.

We used two methods to control for multiple comparisons. Unless otherwise mentioned, the method used was the false discovery rate, with a threshold of q < 0.05 (Genovese, Lazar, & Nichols, 2002). In other cases, we corrected for multiple comparisons using the cluster-size thresholding procedure developed by Forman et al. (1995) extended to 3-D maps and implemented in the Brain Voyager Cluster Threshold plug-in (Goebel, Esposito, & Formisano, 2006). We indicate the use of this method below “corrected alpha < .05.” An initial map was formed using an uncorrected p value of <.005. The minimum cluster size (on the basis of an alpha level of .05) was then set by MonteCarlo simulation using 1000 iterations, simulating the stochastic process of image generation. Afterward, spatial correlations between neighboring voxels were calculated, before voxel intensity thresholds were finally calculated and the corrected map was formed.

ROI Analysis

Given what is known about the striatal components of individual loops, ROIs within the striatum were defined a priori to perform analyses examining changes in activation associated with feedback across scans and for GCM. Striatal ROIs were anatomically defined on the basis of patterns of activation reported in Delgado and colleagues (2005) and Seger and Cincotta (2005) and were drawn to ensure only gray matter within each striatal region was selected and did not extend into any surrounding white matter or gray matter structures such as thalamus or pallidum. Defining ROIs on the basis of anatomy rather than on the basis of the functional scans has the advantage that it avoids the risk of circular reasoning when voxels defined on a functional bases in one analysis are then subjected to additional analyses (Kriegeskorte, Lindquist, Nichols, Poldrack, & Vul, 2010; Kriegeskorte, Simmons, Bellgowan, & Baker, 2009). ROIs were confirmed to meet these conditions for every subject by individually overlaying ROIs onto each subjects' high-resolution, normalized anatomical image. The resulting ROIs were (1) head of the caudate (encompassing 270 voxels centered on x = −10, y = 9, z = 12), (2) body of the caudate (encompassing 516 voxels centered on x = −16, y = −3, z = 20), (3) tail of the caudate (encompassing 372 voxels centered on x = −27, y = −28, z = 2, (4) putamen (encompassing 268 voxels centered on x = −21, y = 3, z = 9), and (5) ventral striatum (encompassing 443 voxels centered on x = −10, y = 7, z = −5). To aid in GCM, a cortical ROI was created for the anterior cingulate gyrus, which is involved in the executive loop and has been thought to continuously interact with the striatum during instrumental learning. (Ullsperger & von Cramon, 2003; Holroyd & Coles, 2002). Unlike primary motor or visual cortices associated with the motor and visual loops, defining the cortical complements of the executive loop proved to be more difficult, given individual differences in anatomy in the pFC. Hence, the anterior cingulate gyrus ROI (encompassing 628 voxels in both hemispheres but centered on x = −1, y = 10, z = 33) was functionally defined. Finally, all ROIs, except the bilateral anterior cingulate ROI, were then translated horizontally across the x axis to create complementary ROIs for the right hemisphere and, again, individually checked to ensure they were located within gray matter and did not include any white matter. The ROI GLM tool of BrainVoyager QX checking for random effects was used to analyze contrasts between conditions separately within each ROI.

GCM

GCM was used to explore effective connectivity between the striatum and other brain regions. This study applied Roebroeck, Formisano, and Goebel's (2005) procedure, as implemented within BrainVoyager, for creating causality maps that provide a measure of directed influence. The ROIs described above were used as reference regions (x) for GCM. Target regions were defined as any voxel not included in the reference region (y). Influence measures were then calculated from the reference to target region (FXY), target to reference region (FYX), and total linear dependence between the reference and target regions (FX,Y) by repeatedly pairing the time course maps of each voxel in these regions. Time course data was sampled from all trials in every condition across three runs. The subsequent GCM analysis was performed using preprocessed data, which included spatial smoothing.

Directed influences to and from the reference region were calculated by subtracting the influence of the target to reference region from the influence from the reference to the target region (FXYFYX) for every voxel to calculate a difference (dGCM). Thus, effective connectivity was described as: dGCM = FXYFYX (see Roebroeck et al., 2005, for details). A positive difference value indicates FXY (reference→volume) influence, whereas negative difference values depict FYX (volume→reference) influence. Statistical significance thresholds for the effective connectivity maps were computed by first creating individual maps for each ROI for each subject, then comparing activation across maps using a voxelwise t test examining whether activity was significantly different from zero, at a cluster threshold of p < .05.

RESULTS

Behavioral Results

Accuracy for probabilistic stimuli was defined in terms of subjects' response according to the dominant category assignment of a stimulus rather than in terms of type of feedback received. As illustrated in Figure 2A, a 2 (Condition: Probabilistic and Random) × 3 (Block) repeated measures ANOVA revealed only a main effect of condition (F(1, 12) = 19.75, p < .01) with higher performance associated with probabilistic trials; the lack of a significant interaction and main effect of block implied that learning occurred early for probabilistic stimuli. To further examine learning during the early stages of the task, one-way repeated measures ANOVA was conducted on probabilistic learning trials within the first run, broken down into quartiles (45 trials each). As illustrated in Figure 2B, this analysis revealed a significant main effect of quartile (F(3, 36) = 6.85, p < .01) as accuracy increased across quartiles. A Bonferonni post hoc test on pairwise comparisons between individual quartiles showed a trend toward a significant difference between the first and second quartiles (p = .06) and a significant difference between the first and third quartiles (p < .01). These results support the notion that probabilistic stimuli were successfully learned during the first scan.

Figure 2. 

Percentage of correct categorization for probabilistic (PROB) and random (RAND) stimuli (A) across all three blocks of learning and (B) within the first block broken down into quartiles. Correct responses were defined in terms of matching the predominant category assignment.

Figure 2. 

Percentage of correct categorization for probabilistic (PROB) and random (RAND) stimuli (A) across all three blocks of learning and (B) within the first block broken down into quartiles. Correct responses were defined in terms of matching the predominant category assignment.

Whole-brain GLM Analyses

For each analysis below, a full list of all activated regions can be found in Supplementary Tables 1–4.

Stimulus–Response Related Activity

To identify neural regions associated with accurate categorization, we compared all probabilistic trials in which stimuli were classified correctly, regardless of feedback received, against trials with incorrect responses (Prob-CP + Prob-CN > Prob-IN) in hopes to further isolate any regions necessary solely for correct categorization, as opposed to regions that are recruited for aspects of the task that are common to both correct and incorrect trials. We found greater activity for correctly classified trials bilaterally in ventrolateral pFC and in the left body of caudate and right head of the caudate (corrected alpha < .05).

Feedback-related Activity

We first examined activation associated with positive feedback by comparing correctly categorized trials resulting in positive feedback (Prob-CP) with baseline. We found large clusters of activation across the striatum including the bilateral head of the caudate nucleus, the anterior putamen, and the tail of the caudate. Activation was also found throughout the pFC (bilateral DLPFC, superior frontal gyrus, and anterior cingulate) and the parietal cortex, mainly the paracentral lobule. Similar patterns of activation were observed for random trials resulting in positive feedback (Rand-P > Baseline). Next, we examined trials resulting in negative feedback (Prob-CN, Prob-IN, and Rand-N > Baseline). Overall, the pattern of activity was similar to that of positive feedback: Increased activation observed in the left head of the caudate, left putamen, and right insula. In addition, there was modulation of activity throughout the anterior and posterior cingulate gyrus and hippocampus. Because of the probabilistic nature of some of the stimulus–category relationships, reward expectations leading up to feedback were different for the various types of trials; positive feedback was expected in some trials and the expectations were violated (Prob-CN), whereas on other trials either negative feedback expected in some trials, or there was no clear expectation (e.g., Prob-IN and Ran-N). When we compared these trials (Prob-CN > Prob-IN + Rand-N), unexpected negative feedback was associated with greater activation in the body of caudate, putamen, ventromedial pFC, and regions of the visual cortex (corrected alpha < .05).

We then compared probabilistic trials on which subjects could have learned and, thus, had feedback expectations, with random trials (Prob-CN + Prob-IN > Rand-N). The probabilistic trials recruited areas throughout the pFC (such as medial prefrontal, anterior cingulate, and DLPFC) as well as regions in the body of the caudate and putamen (corrected alpha < .05).

Stimulus–Response versus Feedback

Because we separated discrete events within each trail via our design, it allowed us to directly contrast stimulus–response and feedback processing. We collapsed across feedback type and reward contingency for all probabilistic trials and treated both the epoch used for the stimulus-locked analyses and the one used for feedback-locked analyses as a predictors. The resulting GLM (Stimulus–Response > Feedback) showed greater striatal activation in the ventral striatum extending into the most inferior portion of the head of the caudate and the posterior caudate (see Figure 3). In addition, we found greater activity throughout the bilateral hippocampus, bilateral motor cortex, bilateral visual cortexes, left DLPFC, superior parietal lobe, and left insula. These patterns of results are consistent with our predictions that the visual loop (posterior caudate and visual cortex activation) and motor loop (motor cortex activation) should be primarily recruited during the stimulus–response phase. Reversing the contrast (Feedback > Stimulus–response) did not yield a significant pattern of activation. Interestingly, some regions of the motivational and executive loops (part of the ventral striatum and pFC, respectively) were actually more active in the stimulus–response phase than the feedback phase. We note that there are executive functions required during stimulus–response phase, in particular, during early learning when subjects tend to evaluate each stimulus.

Figure 3. 

Cortical and striatal areas showing greater activation for stimulus processing/response selection than feedback processing. Left: Sagittal slice illustrating activity in the tail of the caudate and posterior hippocampus (green circle) and the visual cortex. Middle: Axial slice showing activity in the striatum, tail of caudate (green circle), and visual cortex, along with decreased activity in the lateral temporal lobes. Right: Coronal slice showing activity in the ventral striatum, anterior cingulate, and DLPFC. Positive t values are shown in orange–yellow scale.

Figure 3. 

Cortical and striatal areas showing greater activation for stimulus processing/response selection than feedback processing. Left: Sagittal slice illustrating activity in the tail of the caudate and posterior hippocampus (green circle) and the visual cortex. Middle: Axial slice showing activity in the striatum, tail of caudate (green circle), and visual cortex, along with decreased activity in the lateral temporal lobes. Right: Coronal slice showing activity in the ventral striatum, anterior cingulate, and DLPFC. Positive t values are shown in orange–yellow scale.

Effect of Valence: Interaction between Positive and Negative Feedback and Scan

We examined the effects of feedback valence via whole-brain contrasts across all scans and follow up ROI analyses that allowed us to identify interactions between valence and scan. We found no significant main effects of valence in the whole brain analysis in contrasts collapsing across all conditions (Prob-CP + Rand-P > Prob-CN + Prob-IN + Rand-N) or examining only random trials (Ran-P > Ran-N). When only probabilistic trials (Prob-CP > Prob-IN) were examined, there were no significant effects of valence within any striatal region, although there was greater activation throughout the mediofrontal cortex and posterior cingulate gyrus for Prob-CP trials. Although some previous studies have found a main effect of feedback valence within the striatum (Delgado et al., 2005; Seger & Cincotta, 2005), others have found an interaction between valence and stage of learning (Seger et al., 2010). We looked for such an interaction between valence and scan within three dorsal striatal ROIs: head of the caudate, body of the caudate, and putamen. Within the right and left heads of the caudate feedback valence interacted with scan. As shown in Figure 4A, during probabilistic learning trials, activation associated with positive feedback (Prob-CP) remained steadily elevated across scans, whereas activation associated with negative feedback (Prob-IN + Prob-CN) increased across scans. A different pattern was exhibited on random trials (Figure 4B): Activation associated with positive feedback (Rand-P) was greater than for negative feedback (Rand-N) in the first block but the difference between positive and negative feedback converged across the time course of learning. In contrast, there was no interaction between feedback valence and scan in the body of the caudate and the putamen; activation remained significantly elevated above baseline throughout the duration of the task for both negative and positive feedback.

Figure 4. 

Percentage of signal change in the right and left head of the caudate nucleus across scans. (A) Probabilistic trials resulting in positive feedback (Prob-CP) compared with trials resulting in negative feedback (Prob-IN + Prob-CN) trials. (B) Random trials resulting in positive feedback (Rand-P) compared with trials resulting in negative feedback (Rand-N).

Figure 4. 

Percentage of signal change in the right and left head of the caudate nucleus across scans. (A) Probabilistic trials resulting in positive feedback (Prob-CP) compared with trials resulting in negative feedback (Prob-IN + Prob-CN) trials. (B) Random trials resulting in positive feedback (Rand-P) compared with trials resulting in negative feedback (Rand-N).

Granger Causality Analysis

To explore interactions within and between different corticostriatal loops throughout the course of category learning, we measured effective connectivity using GCM, which allowed us to visualize directed influence between striatal and cortical regions. Our measures of effective connectivity were obtained using the anatomically defined ROIs as seed regions. Figure 5 shows the overall patterns of directed influence between striatal regions, and Figure 6 shows patterns of interaction between cortical and striatal regions within each loop. A complete list of areas of directed influence to and from each reference region can be found in Supplementary Table 5.

Figure 5. 

Regions of activation influenced by the reference region (XY) and those regions that influenced the reference region (YX) are shown. Positive values (warm colors) indicate XY influence, whereas negative values (cool colors) depict YX influence. Overall, we observed the putamen exerting directed influence on other parts of the striatum such as the caudate nucleus and ventral striatum. The seed region taken from the anterior caudate exerts directed influence on posterior regions of caudate nucleus, which in turn exerts directed influence further down the tail of the caudate nucleus. Finally, we observed the seed region taken from the ventral striatum receives directed influence from mediofrontal cortex putamen and globus pallidus. For all maps, seed region is shown in magenta.

Figure 5. 

Regions of activation influenced by the reference region (XY) and those regions that influenced the reference region (YX) are shown. Positive values (warm colors) indicate XY influence, whereas negative values (cool colors) depict YX influence. Overall, we observed the putamen exerting directed influence on other parts of the striatum such as the caudate nucleus and ventral striatum. The seed region taken from the anterior caudate exerts directed influence on posterior regions of caudate nucleus, which in turn exerts directed influence further down the tail of the caudate nucleus. Finally, we observed the seed region taken from the ventral striatum receives directed influence from mediofrontal cortex putamen and globus pallidus. For all maps, seed region is shown in magenta.

Figure 6. 

Granger causality maps of cortical regions interacting with striatal seed regions within each corticostriatal loops. Here, we see directed influence within each loop. For example, in the motor loop, the putamen influences regions of the primary motor cortex (column 1). In the executive loop (column 2), the anterior caudate receives directed influence from areas of the medial pFC and exerts directed influence on the DLPFC and superior parietal lobe. In the motivational loop, the ventral striatum receives directed influence from medial and orbito-frontal regions (column 3, top two rows). In the visual loop, the tail of the caudate receives directed influence from the primary visual cortex along the calcarine fissure and higher-order visual areas such as fusiform gyrus (column 3, top two rows) and exerts directed influence on adjoining regions of the tail of the caudate and posterior hippocampus (column 3, bottom two rows). In addition, this figure shows directed influence to and from other cortical regions that suggest interaction between loops. For example, the ventral striatum exerts and receives directed influence to and from visual cortex (column 3, third and fourth rows) and also receives directed influence from areas associated with the executive loop such as the DLPFC (column 3, second rows).

Figure 6. 

Granger causality maps of cortical regions interacting with striatal seed regions within each corticostriatal loops. Here, we see directed influence within each loop. For example, in the motor loop, the putamen influences regions of the primary motor cortex (column 1). In the executive loop (column 2), the anterior caudate receives directed influence from areas of the medial pFC and exerts directed influence on the DLPFC and superior parietal lobe. In the motivational loop, the ventral striatum receives directed influence from medial and orbito-frontal regions (column 3, top two rows). In the visual loop, the tail of the caudate receives directed influence from the primary visual cortex along the calcarine fissure and higher-order visual areas such as fusiform gyrus (column 3, top two rows) and exerts directed influence on adjoining regions of the tail of the caudate and posterior hippocampus (column 3, bottom two rows). In addition, this figure shows directed influence to and from other cortical regions that suggest interaction between loops. For example, the ventral striatum exerts and receives directed influence to and from visual cortex (column 3, third and fourth rows) and also receives directed influence from areas associated with the executive loop such as the DLPFC (column 3, second rows).

Head of the Caudate

As illustrated in Figure 6, second column, the head of the caudate exerted directed influence on the parietal and pFC and received directed influence from several regions along the anterior cingulate gyrus. Within the striatum, the head of the caudate received directed influence from the putamen and exerted directed influence on the body and tail of the caudate, as shown in Figure 5, third column.

Anterior Cingulate

Along with the head of the caudate, we measured effective connectivity to and from a region in the anterior cingulate for a more comprehensive look at patterns of activity within the executive loop. The small seed region within the anterior cingulate exerted directed influence on both head and body/tail regions of the caudate nucleus. In addition, directed influence was observed from prefrontal areas, ventral tegmental area, as well as small clusters of activation along the cingulate gyrus.

Body and Tail of the Caudate

We examined two pairs of ROIs in the posterior caudate, one in the body and one in the tail region. The tail of the caudate ROIs received directed influence from higher-order visual areas in the occipital lobe as well as the fusiform gyrus in the inferior temporal lobe (Figure 6, column 4, rows 1, 2, and 3). In addition, the body of the caudate received directed influence from the anterior cingulate and several other cortical areas such as DLPFC and inferior frontal gyrus (Figure 5, column 3), and the tail of the caudate received directed influence from the motor cortex. Both body and tail regions exerted directed influence on further posterior regions of the posterior caudate and received directed influence from the head of the caudate and putamen. In addition, the tail but not the body of the caudate exerted directed influence on the posterior hippocampus and received directed influence from the thalamus.

Putamen

From our reference regions in the anterior putamen, directed influence was observed in the primary motor cortex (Figure 6, column 1), consistent with the known anatomy of the motor loop, as well as several cortical regions in the parietal and frontal lobes. Within the BG, the putamen exerted directed influence bilaterally to the body of the caudate and pallidum, as seen in Figure 5. We did not find any directed influence into the reference region.

Ventral Striatum

The ventral striatum exerted directed influence on a region of ventromedial frontal cortex (Figure 5, column 4) and received directed influence from the bilateral OFC and hippocampus (Figure 6, column 4, row 4); these patterns of influence are consistent with the known anatomy of the motivational loop. Interestingly, the ventral striatum also exerted directed influence on a very small region of the inferior temporal gyrus, part of the visual loop. Within the BG, the ventral striatum received directed influence from several regions of dorsal striatum, including the head of the caudate and ventral putamen.

DISCUSSION

By dissociating stimulus–response from feedback processing, this study was able to help clarify the shared and individual roles played by corticostriatal loops during visual categorization learning. We found that all the corticostriatal loops were recruited during both stimulus–response and feedback processing, implying that each has a role in both phases of categorization. However, the visual and motor loops were more associated with stimulus–response than feedback processing (as shown in the direct contrast of Stimulus–Response > Feedback), whereas the executive loop was sensitive to feedback processing (as shown in the interaction between feedback valence and scan). Furthermore, GCM revealed both interactions between cortical and striatal regions within individual corticostriatal loops and distinct patterns of influence between loops; in particular, the motor loop exerted influence on the executive and visual loops, and all loops exerted directed influence on the motivational loop.

Visual Loop

Striatal regions within the visual loop, the body and tail of the caudate, were active during both stimulus–response and feedback processing. However, there was greater activation in the tail of the caudate nucleus during the stimulus–response epoch, which is consistent with higher visual categorization demands in this phase of the task. The body was more active for correct than incorrect trials, consistent with it being associated with successful learning (Nomura et al., 2007; Foerde et al., 2006; Seger & Cincotta, 2005, 2006). These findings are consistent with the theory that the posterior caudate, along with regions in the middle and inferior temporal lobe, are necessary for processing complex visual stimuli, and recruitment of this region is important for successful category learning (Seger et al., 2010; Seger, 2008).

GCM showed directed influence between regions within the visual loop. The tail of the caudate received directed influence from higher-order visual areas in both occipital and inferior parietal lobes. Within the visual loop, both seed regions exerted directed influence relatively posterior regions of the caudate. In addition, the body of the caudate received influence from cortical regions involved in the executive and motor loops, including the anterior cingulate gyrus, and prefrontal cortices. This indicates a possible intermediate functional role for the body of the caudate between the executive and visual loops, which is consistent with its anatomical position between the head and tail of the caudate.

Motor Loop

Within the motor loop, the putamen and output nuclei of the BG, such as the pallidum, were active across all conditions in both stimulus–response and feedback phases. Cortical regions involved in the motor loop were more active during stimulus–response than feedback, which is consistent with the greater role of the motor loop in this phase of categorization learning. Interestingly, we did not find an overall difference in putamen activation between phases of categorization learning. The putamen, however, was recruited to different degrees during feedback processing contrasts that differed in whether subjects knew the answer or not; it was more active when subjects were correct (Prob-CN) than when they were incorrect or could not know the answer (Prob-IN and Ran-N) and more active overall when the stimulus–category relationships were learnable (Prob-CN and Prob-IN) than when they were random (Ran-N). These patterns are consistent with the putamen being associated with well-learned associations (Ashby, Turner, & Horvitz, 2010; Daniel & Pollmann, 2010; Seger et al., 2010).

The putamen exerted influence on cortical regions within the motor loop including the motor cortex. During learning, the motor loop is thought to facilitate the selection and execution of appropriate motor responses depending on the task (Seger et al., 2010; Seger, 2008). In addition, the putamen exerted directed influence on regions of the caudate and cortex associated with the executive loop, consistent with an overall influence of the motor loop on the executive loop.

Executive Loop

Within the executive loop, we found recruitment of the anterior caudate nucleus in both stimulus–response and feedback phases of categorization and across all trial types. GCM revealed interactions between striatal and cortical regions within the executive loop. The anterior caudate nucleus exerted directed influence on lateral pFC and superior parietal lobule while receiving influence from the anterior cingulate gyrus.

The response to negative and positive feedback changed across the time course of learning. For probabilistic trials, activity levels for positive and negative feedback were similar in the first scan, but activity associated with positive feedback decreased and activation associated with negative feedback sharply increased across scans. In contrast, for random trials, there was greater activity for positive than negative feedback on Scan 1 that disappeared in subsequent scans. These results indicate that feedback processing in the head of the head of the caudate is not simply a function of feedback. Instead, activity patterns follow the utility of the feedback. In early stages of learning and/or when feedback is random, there is often an overall greater activity for positive feedback, as we found in Scan 1 for the random condition (Delgado et al., 2005; Seger & Cincotta, 2005). However, in situations when negative feedback is less expected and more useful, there is greater activity for negative than positive. The increase in activation associated with negative feedback in the late stages of learning is in line with other tasks involving probabilistic learning or task-switching (Bischoff-Grethe, Hazeltine, Bergren, Ivry, & Grafton, 2009; Monchi et al., 2001), in which negative feedback can become useful signal to indicate a change in rule acquisition strategy.

The pattern of activity we find in the head of the caudate are compatible with studies specifically looking at how the dorsal striatum is involved in acquiring and updating stimulus–response associations that lead to reward (Seger et al., 2010; O'Doherty et al., 2004; Tricomi, Delgado, & Fiez, 2004) via interactions with cortical reward centers (Kringelbach, 2004; Knutson, Fong, Bennett, Adams, & Hommer, 2003). In studies incorporating reinforcement learning modeling, activity in the anterior caudate and putamen is sensitive to “reward prediction” or the expected feedback associated with making the categorical response, whereas the ventral striatum reflects “prediction error” or the difference between expected feedback and actual feedback (Seger et al., 2010).

In addition to the recruitment of the anterior caudate, there were also interesting patterns of activation in cortical regions that interact with this part of the striatum in the executive loop. The ventrolateral pFC was more active for correct than incorrect responses, consistent with theories proposing a role for these region in visual working memory (Goldman-Rakic, 1987) and in mediating between sensory areas and motor areas (Miller, 2000; Fuster, 1997). The anterior cingulate and DLPFC showed greater activity during stimulus–response processing than during feedback.

Motivational Loop

The ventral striatum was significantly more active during stimulus–response than the feedback phase of learning. This pattern runs counter to our initial predictions, which were that the motivational loop should play a primary role at the time of feedback processing. However, this finding complements the results of the GCM, in which we found that the motivational loop received directed influence from all the other corticostriatal loops. These two findings imply that the motivational loop plays a role in integrating all the information about a trial and using it to provide the motivational “set” for the following trial. In this sense, the motivational loop can be seen as a connection between trials, maintaining the history of past reinforcement to be used in the upcoming trials. Previous electrophysiological research in monkeys has found patterns of activity in the ventral striatum consistent with this role in maintaining motivational history or set across trials (Wickens, Budd, Hyland, & Arbuthnott, 2007; Yamada et al., 2007), and researchers have argued that the ventral striatum plays a role in selecting appropriate actions (Nicola, 2007). Interestingly, we found that the ventral striatum exerts a directed influence on visual cortex, which is also consistent with the ventral striatum affecting the functions of the visual loop during stimulus–response processing.

Expectation and Prediction Error in the Striatum

An influential theory that accounts for many (but not all) aspects of striatal function comes from the field of reinforcement learning. Reinforcement learning hypothesizes that agents make predictions about the value of different options (expected reward or reward prediction) and use these values to decide which option to take. After receiving feedback, the agent calculates the difference between the expected and obtained reward, referred to as reward prediction error. Prediction error can then serve as a teaching signal to modify value representations to reflect experience. Reinforcement learning provides a good account of the firing patterns of at least some subpopulations of dopamine neurons from the midbrain (VTA and SN). These neurons project to the whole of the striatum, with densest innervation of the ventro-anteror-medial portions of the striatum, and falling off in intensity across much the same gradient as the anatomical corticostriatal loops. Studies of the BG find that activity in the ventral striatum is well characterized by prediction error, whereas activity in the dorsal striatum, particularly the putamen, is accounted for by reward prediction. The intermediate portions of the corticostriatal loops (head of the caudate/executive loop) have a mixed pattern in which it can be sensitive to both aspects of learning (Haruno & Kawato, 2006; O'Doherty, 2004).

Overall, activity in the head of the caudate nucleus is broadly consistent with the core idea that the striatum represents reward and is particularly sensitive to violations of reward prediction. However, the activation patterns we found are not consistent with a strict model of reinforcement learning and require additional processes to account for them. In this study, subjects learned quickly, and as a result, reward prediction values and prediction error values (when unexpected feedback was obtained) were constant across Blocks 1–3. When negative feedback was received, it was approximately equally likely that it was expected (correctly indicating a mistake) as that it was unexpected. Therefore, if our results were driven primarily by prediction error, we should have found no differences in activity resulting from feedback across blocks. However, as shown in Figure 4, we actually found substantial differences. First, we found for the probabilistic stimuli a striking increase in activity associated with negative feedback on the final block that was not present in Blocks 1 and 2. Reinforcement learning would predict that there should be consistently higher activity for negative feedback over positive across all three blocks. Second, for random stimuli, we found initially higher activity for positive feedback in Block 1 that disappeared in following blocks. Because random stimuli are unlearnable, unexpected positive feedback should be just as likely as unexpected negative feedback, and there should be no valence-based difference and no change across blocks. To account for these differences, it will be necessary to modify traditional reinforcement learning models to include states that can account for cognitive knowledge that goes beyond knowledge of the reward associated with the possible category membership responses for each stimulus. Extensions to reinforcement learning are an active area of research (Zhang, Berridge, Tindell, Smith, & Aldridge, 2009). One promising approach is to add a representation of cognitive state (which can reflect consciously held beliefs or biases) to reinforcement learning models (Glascher, Daw, Dayan, & O'Doherty, 2010).

Interactions between Corticostriatal Loops

Using GCM, we observed that the putamen exerted directed influence on the anterior dorsal caudate nucleus, which in turn exerted directed influence on further posterior regions of the caudate. In addition, both the putamen and dorsal caudate exerted directed influence on the ventral striatum. Thus, the overall pattern we found via the striatal seed regions was putamen > anterior caudate > posterior caudate > ventral striatum or, in terms of loops, motor > executive > visual > motivational. We also found some patterns of directed influence with cortex that were consistent with this pattern. In particular, there was directed influence from visual cortex to the ventral striatum, directed influence from motor cortex to the tail of the caudate, and directed influence from anterior cingulate to the head and body–tail regions of the caudate nucleus.

Overall, the patterns of interaction between loops we observed best fits our functional predictions on the basis of the sequence of events involved in category learning: categorical response (motor) followed by feedback processing (executive) and the updating of reward predictions in preparation for the next trial (motivational). The intermediate position of the executive loop implies that it interacts with both the motivational and motor loops to facilitate the integration of stimulus–response–outcome associations. These interactions are consistent with the open loop projections between these loops described by Joel and Weiner (1994).

Our results are inconsistent with the anatomical gradient from motivational to executive/visual to motor loops (Haber et al., 2000) in two ways. First, the anatomical gradient would imply that the executive loop should affect the motor loop rather than the opposite pattern we found, which was additionally consistent with previous work in our laboratory (Seger et al., 2010). Second, the motivational loop was influenced by all the other loops rather than being the source of all influence: It was at the top of the hierarchy rather than the bottom. However, the motivation loop likely performs both roles: It is the base of the hierarchy in the sense that it holds the motivational context within which the upcoming stimulus is processed and responded to, and in addition, it is at the top of the hierarchy in the sense that it integrates all the information from the trial to serve as the basis for future trials. This dual role is consistent with electrophysiological research, indicating that the ventral striatum can act based on history of past reinforcement to update reward prediction (Wickens et al., 2007; Yamada et al., 2007).

One limitation of GCM in this study is that Granger causality was performed over all trials across three scans; therefore, it is not possible to separate and compare connectivity associated with specific conditions. In addition, the relationships between neural regions revealed by GCM are subject to influence by separate regions not accounted for in the model. Because directed influence is a measure that uses time course values in the seed region to predict values in a target region (and vice versa), directed influence does not take into account the possibility that influence from one region to another may not be direct but rather be mediated by other regions. In addition, another limitation of GCM is that it assumes a homogeneous hemodynamic response in all regions. Because the BOLD response can vary from region to region within the same subject, results from GCM should be interpreted with caution (David et al., 2008). Finally, it should be noted that our GCM analyses do not correct for multiple comparisons and should, thus, be interpreted with caution.

Conclusion

Our results illustrate the dynamic interactions between corticostriatal loops during learning. All loops were recruited during both stimulus–response and feedback processing, emphasizing the need to integrate all three aspects during learning. However, there were some differences in recruitment: The visual and motivational loops were more active during stimulus–response processing than feedback processing, indicating a greater role in determining category membership. The executive loop was sensitive to feedback valence, indicating a greater role in feedback processing. Directed influence progressed from the motor loop to the executive loop, consistent with integration of categorical motor response with its outcome. In addition, both motor and executive loops influenced the motivational loop, providing further evidence that the motivational loop stores history of reward across trials.

Acknowledgments

This research was supported by a grant from the National Institutes of Health (R01MH079182). We thank Erik Peterson, Lucy Troup, and Bruce Draper for thoughtful discussion.

Reprint requests should be sent to Carol A. Seger, Department of Psychology, 1876 Campus Delivery, Colorado State University, Fort Collins, CO 80523, or via e-mail: Carol.Seger@colostate.edu.

REFERENCES

Alexander
,
G. E.
,
DeLong
,
M. R.
, &
Strick
,
P. L.
(
1986
).
Parallel organization of functionally segregated circuits linking BG and cortex.
Annual Review of Neuroscience
,
9
,
357
381
.
Aron
,
A. R.
,
Shohamy
,
D.
,
Clark
,
J.
,
Myers
,
C.
,
Gluck
,
M. A.
, &
Poldrack
,
R. A.
(
2004
).
Human midbrain sensitivity to cognitive feedback and uncertainty during classification learning.
Journal of Neurophysiology
,
92
,
1144
1152
.
Ashby
,
F. G.
,
Alfonso-Reese
,
L. A.
,
Turken
,
A. U.
, &
Waldron
,
E. M.
(
1998
).
A neuropsychological theory of multiple systems in category learning.
Psychological Review
,
105
,
442
481
.
Ashby
,
F. G.
,
Turner
,
B. O.
, &
Horvitz
,
J. C.
(
2010
).
Cortical and basal ganglia contributions to habit learning and automaticity.
Trends in Cognitive Sciences
,
14
,
208
215
.
Bandettini
,
P.
(
2007
).
Functional MRI today.
International Journal of Psychophysiology
,
63
,
138
145
.
Bischoff-Grethe
,
A.
,
Hazeltine
,
E.
,
Bergren
,
L.
,
Ivry
,
R. B.
, &
Grafton
,
S. T.
(
2009
).
The influence of feedback valence in associative learning.
Neuroimage
,
44
,
243
251
.
Buchel
,
C.
,
Holmes
,
A. P.
,
Rees
,
G.
, &
Friston
,
K. J.
(
1998
).
Characterizing stimulus–response functions using nonlinear regressors in parametric fMRI experiments.
Neuroimage
,
8
,
140
148
.
Cincotta
,
C. M.
, &
Seger
,
C. A.
(
2007
).
Dissociation between striatal regions while learning to categorize via feedback and via observation.
Journal of Cognitive Neuroscience
,
19
,
249
265
.
Cools
,
R.
,
Clark
,
L.
,
Owen
,
A. M.
, &
Robbins
,
T. W.
(
2002
).
Defining the neural mechanisms of probabilistic reversal learning using event-related functional magnetic resonance imaging.
Journal of Neuroscience
,
22
,
4563
4567
.
Dale
,
A. M.
(
1999
).
Optimal experimental design for event-related fMRI.
Human Brain Mapping
,
8
,
109
114
.
Daniel
,
R.
, &
Pollmann
,
S.
(
2010
).
Comparing the neural basis of monetary reward and cognitive feedback during information-integration category learning.
Journal of Neuroscience
,
30
,
47
55
.
David
,
O.
,
Guillemain
,
I.
,
Saillet
,
S.
,
Reyt
,
S.
,
Deransart
,
C.
,
Segebarth
,
C.
,
et al
(
2008
).
Identifying neural drivers with functional MRI: An electrophysiological validation.
PLoS Biology
,
6
,
2683
2697
.
Delgado
,
M. R.
,
Locke
,
H. M.
,
Stenger
,
V. A.
, &
Fiez
,
J. A.
(
2003
).
Dorsal striatum responses to reward and punishment: Effects of valence and magnitude manipulations.
Cognitive, Affective & Behavioral Neuroscience
,
3
,
27
38
.
Delgado
,
M. R.
,
Miller
,
M. M.
,
Inati
,
S.
, &
Phelps
,
E. A.
(
2005
).
An fMRI study of reward-related probability learning.
Neuroimage
,
24
,
862
873
.
Delgado
,
M. R.
,
Nystrom
,
L. E.
,
Fissell
,
C.
,
Noll
,
D. C.
, &
Fiez
,
J. A.
(
2000
).
Tracking the hemodynamic responses to reward and punishment in the striatum.
Journal of Neurophysiology
,
84
,
3072
3077
.
Donaldson
,
D. I.
(
2004
).
Parsing brain activity with fMRI and mixed designs: What kind of a state is neuroimaging in?
Trends in Neurosciences
,
27
,
442
444
.
Draganski
,
B.
,
Kherif
,
F.
,
Kloppel
,
S.
,
Cook
,
P. A.
,
Alexander
,
D. C.
,
Parker
,
G. J.
,
et al
(
2008
).
Evidence for segregated and integrative connectivity patterns in the human basal ganglia.
Journal of Neuroscience
,
28
,
7143
7152
.
Ell
,
S. W.
,
Ing
,
A. D.
, &
Maddox
,
W. T.
(
2009
).
Criterial noise effects on rule-based category learning: The impact of delayed feedback.
Attention, Perception & Psychophysics
,
71
,
1263
1275
.
Elliott
,
R.
,
Sahakian
,
B. J.
,
Michael
,
A.
,
Paykel
,
E. S.
, &
Dolan
,
R. J.
(
1998
).
Abnormal neural response to feedback on planning and guessing tasks in patients with unipolar depression.
Psychological Medicine
,
28
,
559
571
.
Fellows
,
L. K.
, &
Farah
,
M. J.
(
2003
).
Ventromedial frontal cortex mediates affective shifting in humans: Evidence from a reversal learning paradigm.
Brain
,
126
,
1830
1837
.
Flaherty
,
A. W.
, &
Graybiel
,
A. M.
(
1995
).
Motor and somatosensory corticostriatal projection magnifications in the squirrel monkey.
Journal of Neurophysiology
,
74
,
2638
2648
.
Foerde
,
K.
,
Knowlton
,
B. J.
, &
Poldrack
,
R. A.
(
2006
).
Modulation of competing memory systems by distraction.
Proceedings of the National Academy of Sciences, U.S.A.
,
103
,
11778
11783
.
Forman
,
S. D.
,
Cohen
,
J. D.
,
Fitzgerald
,
M.
,
Eddy
,
W. F.
,
Mintun
,
M. A.
, &
Noll
,
D. C.
(
1995
).
Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): Use of a cluster-size threshold.
Magnetic Resonance in Medicine
,
33
,
636
647
.
Fuster
,
J. M.
(
1997
).
Network memory.
Trends in Neurosciences
,
20
,
451
459
.
Genovese
,
C. R.
,
Lazar
,
N. A.
, &
Nichols
,
T.
(
2002
).
Thresholding of statistical maps in functional neuroimaging using the false discovery rate.
Neuroimage
,
15
,
870
878
.
Glascher
,
J.
,
Daw
,
N.
,
Dayan
,
P.
, &
O'Doherty
,
J. P.
(
2010
).
States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning.
Neuron
,
66
,
585
595
.
Goebel
,
R.
,
Esposito
,
F.
, &
Formisano
,
E.
(
2006
).
Analysis of functional image analysis contest (FIAC) data with brainvoyager QX: From single-subject to cortically aligned group general linear model analysis and self-organizing group independent component analysis.
Human Brain Mapping
,
27
,
392
401
.
Goldman-Rakic
,
P. S.
(
1987
).
Circuitry of primate prefrontal cortex and regulation of behavior by representational memory.
In V. B. Mountcastle, F. Plum, & S. R. Geiger (Eds.),
Handbook of physiology: The nervous system
(pp.
373
417
).
Bethesda, MD
:
American Physiological Society
.
Groenewegen
,
H. J.
,
Galis-de Graaf
,
Y.
, &
Smeets
,
W. J.
(
1999
).
Integration and segregation of limbic cortico-striatal loops at the thalamic level: An experimental tracing study in rats.
Journal of Chemical Neuroanatomy
,
16
,
167
185
.
Haber
,
S. N.
(
2009
).
Functional anatomy and physiology of the basal ganglia: Non-motor functions.
In D. Tarsy, J. L. Vitek, P. A. Starr, & M. S. Okun (Eds.),
Current clinical neurology: Deep brain stimulation in neurological and psychiatric disorders
(pp.
33
62
).
Totowa, NJ
:
Humana Press
.
Haber
,
S. N.
,
Fudge
,
J. L.
, &
McFarland
,
N. R.
(
2000
).
Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum.
Journal of Neuroscience
,
20
,
2369
2382
.
Haruno
,
M.
, &
Kawato
,
M.
(
2006
).
Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning.
Journal of Neurophysiology
,
95
,
948
959
.
Holroyd
,
C. B.
, &
Coles
,
M. G.
(
2002
).
The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity.
Psychological Review
,
109
,
679
709
.
Joel
,
D.
, &
Weiner
,
I.
(
1994
).
The organization of the basal ganglia-thalamocortical circuits: Open interconnected rather than closed segregated.
Neuroscience
,
63
,
363
379
.
Kasanetz
,
F.
,
Riquelme
,
L. A.
,
Della-Maggiore
,
V.
,
O'Donnell
,
P.
, &
Murer
,
M. G.
(
2008
).
Functional integration across a gradient of corticostriatal channels controls UP state transitions in the dorsal striatum.
Proceedings of the National Academy of Sciences, U.S.A.
,
105
,
8124
8129
.
Knowlton
,
B. J.
,
Mangels
,
J. A.
, &
Squire
,
L. R.
(
1996
).
A neostriatal habit learning system in humans.
Science
,
273
,
1399
1402
.
Knutson
,
B.
,
Fong
,
G. W.
,
Bennett
,
S. M.
,
Adams
,
C. M.
, &
Hommer
,
D.
(
2003
).
A region of mesial prefrontal cortex tracks monetarily rewarding outcomes: Characterization with rapid event-related fMRI.
Neuroimage
,
18
,
263
272
.
Kriegeskorte
,
N.
,
Lindquist
,
M. A.
,
Nichols
,
T. E.
,
Poldrack
,
R. A.
, &
Vul
,
E.
(
2010
).
Everything you never wanted to know about circular analysis, but were afraid to ask.
Journal of Cerebral Blood Flow & Metabolism
,
30
,
1551
1557
.
Kriegeskorte
,
N.
,
Simmons
,
W. K.
,
Bellgowan
,
P. S.
, &
Baker
,
C. I.
(
2009
).
Circular analysis in systems neuroscience: The dangers of double dipping.
Nature Neuroscience
,
12
,
535
540
.
Kringelbach
,
M. L.
(
2004
).
Food for thought: Hedonic experience beyond homeostasis in the human brain.
Neuroscience
,
126
,
807
819
.
Lawrence
,
A. D.
,
Sahakian
,
B. J.
, &
Robbins
,
T. W.
(
1998
).
Cognitive functions and corticostriatal circuits: Insights from Huntington's disease.
Trends in Cognitive Sciences
,
2
,
379
388
.
Lombardi
,
W. J.
,
Andreason
,
P. J.
,
Sirocco
,
K. Y.
,
Rio
,
D. E.
,
Gross
,
R. E.
,
Umhau
,
J. C.
,
et al
(
1999
).
Wisconsin Card Sorting Test performance following head injury: Dorsolateral fronto-striatal circuit activity predicts perseveration.
Journal of Clinical and Experimental Neuropsychology
,
21
,
2
16
.
Maddox
,
W. T.
,
Ashby
,
F. G.
, &
Bohil
,
C. J.
(
2003
).
Delayed feedback effects on rule-based and information-integration category learning.
Journal of Experimental Psychology: Learning, Memory, and Cognition
,
29
,
650
662
.
Maddox
,
W. T.
, &
Ing
,
A. D.
(
2005
).
Delayed feedback disrupts the procedural-learning system but not the hypothesis-testing system in perceptual category learning.
Journal of Experimental Psychology: Learning, Memory, and Cognition
,
31
,
100
107
.
Miller
,
E. K.
(
2000
).
The prefrontal cortex and cognitive control.
Nature Reviews Neuroscience
,
1
,
59
65
.
Monchi
,
O.
,
Petrides
,
M.
,
Petre
,
V.
,
Worsley
,
K.
, &
Dagher
,
A.
(
2001
).
Wisconsin Card Sorting revisited: Distinct neural circuits participating in different stages of the task identified by event-related functional magnetic resonance imaging.
Journal of Neuroscience
,
21
,
7733
7741
.
Nicola
,
S. M.
(
2007
).
The nucleus accumbens as part of a basal ganglia action selection circuit.
Psychopharmacology (Berlin)
,
191
,
521
550
.
Nomura
,
E. M.
,
Maddox
,
W. T.
,
Filoteo
,
J. V.
,
Ing
,
A. D.
,
Gitelman
,
D. R.
,
Parrish
,
T. B.
,
et al
(
2007
).
Neural correlates of rule-based and information-integration visual category learning.
Cerebral Cortex
,
17
,
37
43
.
O'Doherty
,
J.
,
Dayan
,
P.
,
Schultz
,
J.
,
Deichmann
,
R.
,
Friston
,
K.
, &
Dolan
,
R. J.
(
2004
).
Dissociable roles of ventral and dorsal striatum in instrumental conditioning.
Science
,
304
,
452
454
.
O'Doherty
,
J. P.
(
2004
).
Reward representations and reward-related learning in the human brain: Insights from neuroimaging.
Current Opinion in Neurobiology
,
14
,
769
776
.
Passingham
,
R. E.
(
1993
).
The frontal lobes and voluntary action
.
New York
:
The Clarendon Press
.
Poldrack
,
R. A.
,
Clark
,
J.
,
Pare-Blagoev
,
E. J.
,
Shohamy
,
D.
,
Creso Moyano
,
J.
,
Myers
,
C.
,
et al
(
2001
).
Interactive memory systems in the human brain.
Nature
,
414
,
546
550
.
Poldrack
,
R. A.
,
Prabhakaran
,
V.
,
Seger
,
C. A.
, &
Gabrieli
,
J. D.
(
1999
).
Striatal activation during acquisition of a cognitive skill.
Neuropsychology
,
13
,
564
574
.
Roebroeck
,
A.
,
Formisano
,
E.
, &
Goebel
,
R.
(
2005
).
Mapping directed influence over the brain using Granger causality and fMRI.
Neuroimage
,
25
,
230
242
.
Seger
,
C. A.
(
2008
).
How do the basal ganglia contribute to categorization? Their roles in generalization, response selection, and learning via feedback.
Neuroscience and Biobehavioral Reviews
,
32
,
265
278
.
Seger
,
C. A.
, &
Cincotta
,
C. M.
(
2005
).
The roles of the caudate nucleus in human classification learning.
Journal of Neuroscience
,
25
,
2941
2951
.
Seger
,
C. A.
, &
Cincotta
,
C. M.
(
2006
).
Dynamics of frontal, striatal, and hippocampal systems during rule learning.
Cerebral Cortex
,
16
,
1546
1555
.
Seger
,
C. A.
, &
Miller
,
E. K.
(
2010
).
Category learning in the brain.
Annual Review of Neuroscience
,
33
,
203
219
.
Seger
,
C. A.
,
Peterson
,
E. J.
,
Cincotta
,
C. M.
,
Lopez-Paniagua
,
D.
, &
Anderson
,
C. W.
(
2010
).
Dissociating the contributions of independent corticostriatal systems to visual categorization learning through the use of reinforcement learning modeling and granger causality modeling.
Neuroimage
,
50
,
644
656
.
Shohamy
,
D.
,
Myers
,
C. E.
,
Kalanithi
,
J.
, &
Gluck
,
M. A.
(
2008
).
Basal ganglia and dopamine contributions to probabilistic category learning.
Neuroscience and Biobehavioral Reviews
,
32
,
219
236
.
Talairach
,
J.
, &
Tournoux
,
P.
(
1998
).
Co-planar stereotaxic atlas of the human brain: 3-Dimensional proportional system—An approach to cerebral imaging
.
New York
:
Thieme Medical Publishers
.
Tricomi
,
E. M.
,
Delgado
,
M. R.
, &
Fiez
,
J. A.
(
2004
).
Modulation of caudate activity by action contingency.
Neuron
,
41
,
281
292
.
Ullsperger
,
M.
, &
von Cramon
,
D. Y.
(
2003
).
Error monitoring using external feedback: Specific roles of the habenular complex, the reward system, and the cingulate motor area revealed by functional magnetic resonance imaging.
Journal of Neuroscience
,
23
,
4308
4314
.
Voorn
,
P.
,
Vanderschuren
,
L. J.
,
Groenewegen
,
H. J.
,
Robbins
,
T. W.
, &
Pennartz
,
C. M.
(
2004
).
Putting a spin on the dorsal-ventral divide of the striatum.
Trends in Neurosciences
,
27
,
468
474
.
Wickens
,
J. R.
,
Budd
,
C. S.
,
Hyland
,
B. I.
, &
Arbuthnott
,
G. W.
(
2007
).
Striatal contributions to reward and decision making: Making sense of regional variations in a reiterated processing matrix.
Annals of the New York Academy of Sciences
,
1104
,
192
212
.
Williams
,
Z. M.
, &
Eskandar
,
E. N.
(
2006
).
Selective enhancement of associative learning by microstimulation of the anterior caudate.
Nature Neuroscience
,
9
,
562
568
.
Yamada
,
H.
,
Matsumoto
,
N.
, &
Kimura
,
M.
(
2007
).
History- and current instruction-based coding of forthcoming behavioral outcomes in the striatum.
Journal of Neurophysiology
,
98
,
3557
3567
.
Zhang
,
J.
,
Berridge
,
K. C.
,
Tindell
,
A. J.
,
Smith
,
K. S.
, &
Aldridge
,
J. W.
(
2009
).
A neural computational model of incentive salience.
PLoS Computational Biology
,
5
,
e1000437
.
Zheng
,
T.
, &
Wilson
,
C. J.
(
2002
).
Corticostriatal combinatorics: The implications of corticostriatal axonal arborizations.
Journal of Neurophysiology
,
87
,
1007
1017
.