We test theories about the functional organization of the human cortex by correlating brain activity with demands on perception versus action selection. Subjects covertly searched for a target among an array of 4, 8, or 12 items (perceptual manipulation) and then, depending on the color of the array, made a saccade toward, away from, or at a right angle from the target (action manipulation). First, choice response times increased linearly as the demands increased for each factor, and brain activity in several cortical areas increased with increasing choice response times. Second, we found a double dissociation in posterior cortex: Activity in ventral regions (occipito-temporal cortex) increased linearly with perceptual, but not action, selection demands; conversely, activity in dorsal regions (parietal cortex) increased linearly with action, but not perceptual, selection demands. This result provides the clearest support of the theory that posterior cortex is segregated into two distinct streams of visual processing for perception and action. Third, despite segregated anatomical projections from posterior ventral and dorsal streams to lateral pFC, we did not find evidence for a functional dissociation between perception and action selection in pFC. Increasing action, but not perceptual, selection demands evoked increased activation along both the dorsal and the ventral lateral pFC. Although most previous studies have focused on perceptual variables (e.g., space vs. object), these data suggest that understanding the computations underlying action selection will be key to understanding the functional organization of pFC.
The primate visual system is divided into two distinct anatomical pathways: a ventral pathway that projects from visual cortex to the temporal lobe and a dorsal pathway that projects from visual cortex to the posterior parietal lobe (Ungerleider & Mishkin, 1982). Visual information is transformed for object perception in the ventral pathway and for action selection in the dorsal pathway (Goodale & Milner, 1992; Ungerleider & Mishkin, 1982; but see also Cusack, Mitchell, & Duncan, 2010; Konen & Kastner, 2008). Lesions of the ventral pathway selectively impair visual perception (e.g., visual-form agnosia; Farah, 1990), whereas lesions of the dorsal pathway selectively impair spatial attention and the visual control of action (e.g., optic ataxia; Battaglia-Mayer & Caminiti, 2002). Previous neuroimaging studies on healthy humans have reported that the posterior ventral cortex is activated in perceptual tasks, such as object recognition (Kourtzi & Kanwisher, 2000), whereas posterior parietal cortex (PPC) is activated in spatial attention and movement tasks (Ikkai & Curtis, 2008; Culham et al., 2003). These studies indicate that the posterior ventral and dorsal pathways have distinct roles in visual information processing.
The two posterior pathways continue as largely segregated afferents to the frontal lobes, forming a ventral occipito-temporo–prefrontal pathway and a dorsal occipito-parieto–prefrontal pathway (Averbeck & Seo, 2008; Romanski, 2004; Macko et al., 1982). What is the fate of these segregated streams of processing in the frontal lobes? Theories are mostly agnostic, and relevant data are scant on this issue. Goldman-Rakic (1996) extrapolated from the specialized object processing in the ventral “what” pathway and spatial processing in the dorsal “where” pathway of posterior cortex (Ungerleider & Mishkin, 1982) to the ventral and dorsal lateral pFC, respectively. Her influential “domain specificity” hypothesis proposed that the lateral pFC is specialized for working memory, with the ventrolateral pFC (VLPFC) and dorsolateral pFC (DLPFC) maintaining the type of information that they receive from posterior pathways; that is, VLPFC maintains nonspatial information from the “what” pathway and DLPFC maintains spatial information from the “where” pathway. Consistent with this hypothesis, studies have reported object or cue-related selectivity in VLPFC (Scalaidhe, Wilson, & Goldman-Rakic, 1999; Courtney, Petit, Maisog, Ungerleider, & Haxby, 1998; Wilson, Scalaidhe, & Goldman-Rakic, 1993) and spatial selectivity in DLPFC (Funahashi, Bruce, & Goldman-Rakic, 1989, 1993). However, other studies either did not find a spatial–nonspatial dissociation (Postle, Berger, Taich, & D'Esposito, 2000; Postle & D'Esposito, 1999; Owen, Evans, & Petrides, 1996) or proposed an altogether different functional architecture of pFC (Badre & D'Esposito, 2007; Koechlin, Ody, & Kouneiher, 2003; Fuster, 2001; D'Esposito, Postle, & Rypma, 2000; D'Esposito, Postle, Ballard, & Lease, 1999). Moreover, lesions of the ventral and dorsal pFC do not provide conclusive support for the domain specificity model (Curtis & D'Esposito, 2004). Indeed, the presence, if any (Duncan & Owen, 2000), of a functional parcellation of pFC is a current topic of intensive investigation in cognitive neuroscience (Badre & D'Esposito, 2009; Badre, 2008; Botvinick, 2008). Surprisingly, previous studies and theory have largely ignored motor factors, despite the fact that pFC is thought to be at the apex of the motor hierarchy (Fuster, 2001).
Here, we test for a double dissociation between perception and action selection in the ventral and dorsal pathways of the human cortex. We predict that if the posterior visual streams can be clearly segregated according to perceptual and motor functions, then this functional organization may propagate to the lateral pFC. We test this hypothesis with a unique visuomotor search task that parametrically manipulates the demands on perception and action selection.
Nineteen neurologically healthy subjects (5 men, 18 right-handed, age between 18 and 39 years) were recruited for participation and were paid for their time. All subjects had normal or corrected-to-normal vision. Subjects gave written informed consent, and all procedures were in compliance with the safety guidelines for fMRI research and approved by the human subjects institutional review board at New York University. One subject's data were discarded from the analyses because of excessive movement (>10 mm) during fMRI data acquisition.
Behavioral Procedures and Factorial Design
The experimental stimuli were controlled by E-Prime (Psychology Software Tools, Inc., Pittsburgh, PA) and projected (Eiki LC-XG100, Rancho Santa Margarita, CA) into the bore of the scanner on a screen that was viewed by the subjects through an angled mirror. Subjects fixated a central white cross against a black background until a search array was presented. Examples of three of the nine possible search array displays are shown in Figure 1A. Search arrays consisted of one target (letter “T”) and 3, 7, or 11 distractors (letters “L”), both of which were in Arial Black font to increase the target–distractor similarity and thus to induce inefficient search slope. Both were 1.5° of visual angle high and wide and presented within an invisible annulus with an outer radius of 5.75° of visual angle. Search arrays were visible for 3 sec while subjects covertly searched for the target (i.e., gaze remained at fixation). Target and distractors could be presented in any of three colors (yellow, magenta, and cyan) and in any of four orientations (0°, 90°, 180°, and 270°). A variable intertrial interval (3, 5, 7, 9, or 11 sec) was used between search trials.
We used a fully crossed factorial design with two factors, perceptual selection and action selection, each with three levels yielding a total of nine trial types. To increase the demands on perceptual selection processes, we manipulated the set size of the array. The arrays consisted of 4, 8, or 12 letter items. Placeholders (dots, 0.6° of visual angle) were used to populate the entire array to approximate an equal visual display (e.g., visual luminance, spatial coverage). Letters and placeholders were arranged so that each quadrant contained one letter and two placeholders in Set Size 4 trials and two letters and one placeholder in Set Size 8 trials. Only letters were used in Set Size 12 trials. To increase the demands on action selection, we manipulated the spatial compatibility between the location of the target and the required saccade (i.e., saccade transformation). The color of the search array specified the saccade transformation (Figure 1A). The color instructed subjects to look to the target (prosaccade), 180° opposite from the target (antisaccade), or rotated clockwise or counterclockwise 90° from the target (rotation saccade). The color-saccade transformation assignment was counterbalanced across subjects, and the order of trial types was pseudorandomized. The location of the target and the color of the search array were pseudorandomized so that neither the search nor the saccade target appeared in the same place on more than two trials in a row, and the same color did not repeat on more than two trials. Each scanning session consisted of eight blocks of four trials per condition, yielding a total of 36 trials per block and a total of 32 trials per condition in a scanning session. One subject terminated the experiment after five blocks (20 trials per condition). Subjects were trained on the task outside the scanner on the day before the scanning session (∼1 hour).
Eye position was monitored in the scanner at 60 Hz with an infrared videographic camera equipped with a telephoto lens (ASL 504LRO; Applied Sciences Laboratories, Bedford, MA; modified with a Sony HAD CCD) that focused on the right eye viewed from the flat surface mirror mounted inside the RF coil. Nine-point calibrations were performed at the beginning of the session and between blocks when necessary. Eye movement data were transformed to degrees of visual angle, calibrated using a third-order polynomial algorithm that fit eye positions to known spatial positions and scored off-line with in-house software (GRAPES). Any trials with unwanted/incorrect saccades were discarded (e.g., overt search, corrective saccade, saccade to wrong item). Only trials in which the first saccade landed on the correct target and remained there until the search array offset was further analyzed. Saccadic RTs were estimated with semiautomatic routines that relied on the velocity of the eye reaching about 30°/sec to determine the onset of saccades. The data were also inspected visually, trial by trial, and corrections were made if necessary.
fMRI data were collected using a 3-T head-only scanner (Allegra, Siemens, Erlangen, Germany) at the Center for Brain Imaging at New York University. Images were acquired using custom radio-frequency coils (NM-011 transmit head coil and NMSC-021 four-channel phased array receive coil; NOVA Medical, Wakefield, MA) placed over lateral frontal and parietal cortices. During each fMRI scan, a series of volumes was acquired using a T2*-sensitive EPI pulse sequence (repetition time [TR] = 2000 msec, echo time = 30 msec, flip angle = 80°, 36 slices, 3 × 3 × 3-mm voxels, field of view = 192 × 192 mm). High-resolution (1-mm isotropic voxels) MP-RAGE three-dimensional T1-weighted scans were acquired for anatomical registration, segmentation, and display. To minimize head motion, we stabilized subjects with foam padding around the head.
BOLD Analytic Procedures
Post hoc image registration was used to correct for residual head motion (MCFLIRT; motion correction using the linear image registration tool from Oxford University's Center for Functional MRI of the Brain; Jenkinson, Bannister, Brady, & Smith, 2002). Additional preprocessing of the fMRI data was as follows. First, we band-pass filtered the time series of each voxel (0.05–0.25 Hz) to compensate for the slow drift typical in fMRI measurements (Zarahn, Aguirre, & D'Esposito, 1997) and divided the time series of each voxel by its mean intensity to convert to percent signal modulation and compensate for the decrease in mean image intensity with distance from the receive coil.
The fMRI response was modeled with an impulse time locked to the onset of the search array convolved with a canonical hemodynamic response function (Polonsky, Blake, Braun, & Heeger, 2000). Each level of both factors (Set Size 4—prosaccade, Set Size 4—antisaccade,…, Set Size 12—rotation saccade, etc.) was modeled separately in the design matrix and entered into a modified general linear model (Worsley & Friston, 1995) for statistical analysis using VoxBo (http://www.voxbo.org). For each subject, we used the software Caret (http://brainmap.wustl.edu/caret) for anatomical segmentation, gray-white matter surface generation, flattening, and multifiducial deformation mapping to the PALS atlas (Van Essen, 2005). Registering subjects in a surface space using precise anatomical landmark constraints (e.g., central sulcus, sylvian and calcarine fissures, etc.) results in greater spatial precision of the alignment compared with standard volumetric normalization methods. Statistical maps for contrasts of interest were created using the beta-weights estimated from each subject's general linear model. For overall task-related activity, we contrasted all trial types with the intertrial interval baseline. To estimate linear increases in activation, we used weights of −1, 0, +1 to model increasing task demands associated with the appropriate levels of set size and saccade transformation. These parameter maps were then deformed into the same atlas space, and t statistics were computed for each contrast across subjects in spherical atlas space. We used a nonparametric statistical approach on the basis of permutation tests to help address the problem of multiple statistical comparisons, which are even more problematic when one performs statistical analyses on surfaces. First, we constructed a permuted distribution of clusters of neighboring surface nodes with t values > 2.5. We chose a primary t statistic cutoff of 2.5 because it is strict enough that intense focal clusters of activity would pass but not so strict that diffuse large clusters of activity are lost. In the case of a one-sample comparison, where measured values are compared with the test value of 0, the signs of the beta values for each node were randomly permuted for each subject's surface, before computing the statistic. One thousand iterations, N, of this procedure were performed to compute a permutation distribution for each statistical test performed. Then, we ranked the resulting suprathreshold clusters by their area. Finally, corrected p values at α = .05 for each suprathreshold cluster were obtained by comparing their area to the area of the top 5% of the clusters in the permuted distribution. Where C = Nα + 1, we considered clusters ranked Cth or smaller significant at t = 2.5. The permutation tests controlled for type I error by allowing us to formally compute the probability that an activation of a given magnitude could cluster together by chance (Nichols & Holmes, 2001; Holmes, Blair, Watson, & Ford, 1996).
To examine the relationship between measured behavior (RT) and neural activity (BOLD signal), we computed statistical maps that reflected correlations between evoked BOLD activity and saccade RTs on a trial-by-trial basis. To do so, excluding incorrect trials, we regressed RT (convolved with a hemodynamic response function) against BOLD time courses. The resulting beta maps from each subject were entered into a second-level analysis in which we estimated the reliability of the effects across subjects (N = 18) with a t statistic (corrected for multiple comparisons as described above).
Time Series Analytic Procedures
We used ROI-based analyses of the time courses of BOLD signal change. First, on each subject's high-resolution anatomical scans, we traced around gray matter of several ROIs motivated by past studies of visual search and oculomotor processing (Curtis & Connolly, 2008; Ikkai & Curtis, 2008; Jerde et al., 2008; Srimal & Curtis, 2008) and preliminary inspection of task-related deviations from baseline for all trial types in single subjects. In posterior cortex, these areas included the collateral sulcus (CoS; along ventromedial surface of occipital to temporal lobes), the horizontal segment of the intraparietal sulcus (hIPS; extended from the junction with postcentral sulcus to the junction with parieto-occipital sulcus), the superior parietal lobule (SPL; the gyrus posterior to the postcentral sulcus, anterior to the parietal-occipital sulcus, and lateral to the interhemispheric sulcus, medial to hIPS), and the inferior parietal lobule (IPL; the gyrus posterior to the postcentral sulcus, anterior to the parietal-occipital sulcus, and lateral to the hIPS). The IPL was chosen because of its unique negative activation during all trial types (Figure 2A).
In the lateral pFC, ROIs included the superior precentral sulcus (sPCS; along the precentral sulcus and lateral to the junction with superior frontal sulcus [SFS]), the SFS, the inferior frontal sulcus (IFS), and the dorsal and ventral middle frontal gyrus (dMFG and vMFG, respectively). SFS and IFS were defined as the gray matter within these sulci but not extending past the gyral convexity. Dorsal and ventral MFG were separated by the fundus of the intermediate frontal sulcus. As can be seen in Figure 4A, SFS, dMFG, vMFG, and IFS divide pFC into four ROIs from dorsal to ventral along the lateral pFC. We were not able to identify the intermediate frontal sulcus in one of the subjects. For this subject, all ROIs but dMFG and vMFG were included in the analysis, which led to n = 17 for dMFG and vMFG and n = 18 for all other ROIs. Within each ROI, we used an F test to select 20 voxels (540 mm3) with the strongest overall task effect. These voxels showed a consistent deviation from baseline during the task. The selection is unbiased by activation (could be negative or positive relative to baseline) or trial type (none of the 2 factors × 3 levels are given unique weight). BOLD data were converted into percent signal change, and time courses time locked to the onset of the search array were deconvolved using AFNI (http://afni.nimh.nih.gov/afni), with no hemodynamic response assumed. The estimated impulse response functions were averaged across voxels within an ROI and averaged across subjects from analogous ROIs to visualize time series. Error bars are standard deviations between subjects at each time point. For an individual subject, the average of three TRs around the peak of the impulse response function (time points 4, 6, and 8 sec) from each condition was extracted from each ROI and used as a dependent variable in statistical analyses of the time courses.
Retinotopic Mapping Methods and Analyses
In four subjects, we identified subregions within the visual cortex with standard retinotopic mapping procedures (Larsson & Heeger, 2006; Sereno et al., 1995; Engel et al., 1994). Stimuli were controlled by MGL (http://justingardner.net/mgl), and stimulus presentation and analysis were performed following procedures in Gardner, Merriam, Movshon, and Heeger (2008) and Larsson and Heeger (2006). During the scanning session, observers viewed high-contrast checkerboard stimuli presented within a rotating wedge aperture while fixating the center of the display. Wedge apertures subtended 90° of polar angle and extended from 0.6° to 12.6° eccentricity, moving in the stepwise increment of 10° visual angle per TR. Regions outside the aperture were gray. Each scan lasted 252 sec, yielding 10.5 cycles, but we discarded the first half cycle of each scan before analysis. All subjects completed at least six scans of wedges (three clockwise and three counterclockwise). One subject had five and four clockwise and counterclockwise scans, respectively, and another subject had four clockwise and counterclockwise scans. The time series for each scan were coarsely corrected for hemodynamic delay by shifting the time series of each voxel by 4 sec. The time series for the counterclockwise wedges were then time reversed and averaged with the time series for clockwise wedges. Phase information was then projected onto the flattened surface generated in Caret (Van Essen, 2005). Here, we identified V1, dorsal and ventral V2, and V3 boundaries using phase reversals in the angular component of the visual field representation. These ROIs were then transformed back to each subject's functional space, and time series were extracted.
As predicted, subject performance was better when the set size was smaller and when the saccade transformation was simpler. Across trial types, subjects were on average 86% accurate. A repeated measures ANOVA of accuracy revealed significant linear effects of set size, F(1, 17) = 22.10, p < .001, effect size = 0.57, and saccade transformation (prosaccade < antisaccade < rotation saccade), F(1, 17) = 22.65, p < .001, effect size = 0.59, but critically the two factors did not interact, F(4, 68) = 0.69, p > .05. Although a different number of trials contributed to each condition, our post hoc power analysis revealed large power for the linear analysis of set size (power = .991) and saccade transformation (power = .993). Similarly, RTs increased linearly as the set size increased, F(1, 17) = 179.49, p < .001, effect size = 0.91, and as the saccade transformation became more difficult, F(1, 17) = 87.28, effect size = 0.84. Neither factor showed a quadratic effect (p values > 0.05). Figure 1B shows the RT data from a representative subject, and Figure 1C shows the average normalized RT of all subjects. The slope for the set size effect collapsed across saccade transformation was 40 msec/item, and the slope for the saccade transformation effect collapsed across set size was 211 msec/transformation (prosaccade, antisaccade, and rotation saccade). Univariate ANOVAs within each subject revealed that the main effect of set size was statistically significant in all subjects (all p values < .003); RT was fastest in Set Size 4 trials, followed by Set Size 8, and slowest in Set Size 12 (linear contrast, all p values < .003). All but two subjects (16/18) showed a significant main effect of saccade transformation; saccade initiation was fastest for prosaccades, followed by antisaccades, and slowest for rotation saccades. One of the two subjects who did not show a main effect of saccade transformation approached significance (p = .08). The interaction of set size and saccade transformation on RT was not significant in all but two subjects. Overall, our manipulations of perceptual selection (set size) and action selection (saccade transformation) had robust, reliable, and predictable effects on behavioral performance that make interpreting the neuroimaging data straightforward.
Imaging Results: Cortical Surface Statistical Analyses
Performance of the visuomotor search task evoked BOLD activity in occipital, posterior temporal, parietal, and a large extent of frontal cortices bilaterally (Figure 2A). Moreover, we found robust positive correlations between behavioral RTs and bilateral activation in these same areas (occipital and parietal cortices, superior and inferior precentral sulcus [sPCS and iPCS, respectively], insula, and pre-SMA; Figure 3), indicating a strong coupling of neural activation with task performance. In addition, right central sulcus, right inferior frontal gyrus, left intermediate sulcus, and left IFS showed positive correlations with the RT. We found a significant negative correlation in the right IPL. Positive correlations may reflect the greater neural activity (duration or magnitude) associated with the more demanding level of the factor (e.g., saccade transformation: rotation saccade > antisaccade > prosaccade) that resulted in longer RTs. These data nicely link theory (experimental conditions), measured behavior (RT), and neural activity (BOLD signal).
In posterior cortex, we observed a clear distinction between areas showing a linear effect of set size and of saccade transformation (Figure 2B). The linear effect of set size (Set Size 12 > Set Size 4, Figure 2B, red) was observed bilaterally in the area extending from the occipital lobe to the medial surface of the temporal lobe (p < .001, corrected), whereas the linear effect of saccade transformation (rotation saccade > prosaccade, Figure 2B, green) was observed in bilateral posterior-to-anterior parietal lobe (p < .001, corrected). A negative linear effect of saccade transformation was observed in the right IPL (p < .03, corrected) and left posterior cingulate (p < .03, corrected). We also observed an overlap between these linear effects at the lateral side of the occipital cortex (Figure 2B, yellow). This region, located just posterior to the parieto-occipital sulcus in the right hemisphere, could be sensitive to both set size and saccade transformation, thus making it a bridge between visual processing for perception and action. Overall, these data show a clear functional double dissociation between perceptual selection and action selection in the ventral and dorsal regions of the posterior human brain, respectively.
We did not find a double dissociation in the frontal cortex, where no clusters showed a significant linear effect of array set size (Figure 2C). However, the left precentral sulcus (p = .003, corrected) showed a linear effect of saccade transformation. In the right hemisphere, the effect was marginal along the precentral sulcus (p = .103 and .068 for the superior and inferior precentral sulcus, corrected), but the pattern was similar to that of the left hemisphere.
To summarize, the cortical surface statistics revealed a clear distinction between posterior ventral and dorsal areas as the demands on perceptual selection and action selection increased, respectively. Conversely, such effects were not observed in frontal areas. We considered the possibility that this result may be partly due to poor intersubject anatomical coregistration. Although we used inflated surface-based registration to ensure higher intersubject alignment consistency (Van Essen, 2005), registration in frontal cortex is generally poor compared with posterior cortex due to large individual differences in sulcal folding patterns. Evidence for individual variability is exemplified by the fact that one subject did not have an intermediate frontal sulcus. To minimize these effects, we performed time series analyses on ROIs on the basis of anatomical criteria applied on an individual subject basis. With this approach, we parcellated the lateral frontal cortex into four anatomical ROIs spanning the dorsal to ventral convexity. Namely, we identified the SFS, dMFG, vMFG, and IFS. For comparison purposes, we included additional ROIs that included the CoS, hIPS, SPL, IPL, and sPCS. Only the right hemisphere ROI was analyzed for the IPL because of its unique negative activation during the task versus baseline contrast (Figure 2A).
Imaging Results: ROI Time Series Analyses
In three of our ROIs (CoS, hIPS, and sPCS), right hemisphere activation was larger than left hemisphere activation: CoS, F(1, 17) = 6.586, p < .03; hIPS, F(1, 17) = 8.818, p < .01; sPCS, F(1, 17) = 7.790, p < .03. Therefore, the right and left hemispheres were analyzed separately for these ROIs. It is important to note that although the right hemisphere activation was overall larger in these ROIs, the same patterns of time series (Figure 4B and C) and linear effects (see below) were observed in the right and left hemispheres. For the other ROIs that did not show an effect of hemisphere, we collapsed across hemispheres to increase statistical power. Figure 4 plots the subject-averaged time series from each of the ROIs, time locked to the onset of the search array. Data from the right hemisphere (“r−”) are shown for CoS, hIPS, and sPCS. As a dependent variable, we averaged the signal from the three time points around the peak of the hemodynamic response for each subject (see Methods; gray epochs in Figure 4B and C). To be completely consistent in our ROI criteria, we omitted the one subject who did not have an intermediate frontal sulcus from the analyses involving the dMFG and vMFG.
In ventral posterior cortex, CoS (Figure 4B) showed a significant linear increase in the BOLD signal as the set size increased [left: F(1, 17) = 4.80, p < .05; right: F(1, 17) = 9.77, p < .01], but no effect of saccade transformation (both F values < 1, p > .05). In four subjects, we collected additional BOLD data to create retinotopic maps of visual cortex in the occipital lobe (Figure 5A). Similar to anatomically defined CoS, we found a significant linear effect of set size in V2d, F(1, 3) = 12.94, p < .05, V2v, F(1, 3) = 12.10, p < .05, and V3d, F(1, 3) = 14.97, p < .05 (Figure 5B). This effect was not reliable but approached significance in V1, F(1, 3) = 7.63, p = .070, and V3v, F(1, 3) = 7.86, p = .068. None of the retinotopically defined ROIs showed a significant linear effect of saccade transformation (all F values < 1, all p values > .05).
In dorsal posterior cortex, hIPS and SPL showed a significant linear increase as the saccade transformation demands increased [right hIPS: F(1, 17) = 75.77, p < .001; left hIPS: F(1, 17) = 77.47, p < .001; SPL: F(1, 17) = 64.88, p < .001], but no effect as set size increased (all F values < 1, all p values > .05) (Figure 4C). Thus, consistent with the surface statistics, the time series extracted from posterior ventral and dorsal ROIs showed a double dissociation: BOLD responses in occipital and temporal cortex increased significantly with greater set size, whereas activation in PPC increased significantly with a greater demand on saccade transformation. A Region × Condition interaction between hIPS and CoS was significant for a linear effect of set size, F(1, 17) = 11.19, p < .005, and saccade transformation, F(1, 17) = 43.89, p < .001.
The right IPL was the only cortical region across both hemispheres that was negatively activated during the task, but neither linear effect of set size nor saccade transformation was significant, F(1, 17) < 1.12, p > .05. There was a significant quadratic effect of set size, F(1, 17) = 5.37, p < .05. Post hoc paired t test revealed that the quadratic effect was driven by the significant difference between Set Size 4 and Set Size 8, t(17) = 2.35, p < .05.
Similar to the time courses from PPC, pFC ROIs (left and right sPCS, SFS, vMFG, and IFS) showed a significant linear increase in activation for saccade transformation (all F values > 8.0, all p values < .03), except dMFG, F(1, 16) = 3.38, p = .08 (Figure 4C). None of pFC ROIs showed an effect of set size (all F values < 0.33). Consistent with the behavioral data, no ROI showed an interaction between set size and saccade transformation or a quadratic effect of either factor.
We used rapid event-related fMRI to measure cortical activity during a visual search task that parametrically manipulated the demands on perceptual and action selection. Previous fMRI studies have reported a dissociation between posterior ventral and dorsal regions for perception and action, but the tasks involved action recognition (Shmuelof & Zohary, 2005), not action itself, or examined perception and action in separate experimental conditions and subsequently compared the resulting activation across regions (Cavina-Pratesi, Goodale, & Culham, 2007). The present study used a single search task that manipulated the demands on perception and action selection in a factorial design (Figure 1). Consistent with past behavioral studies, RTs increased as the number of potential targets increased (Treisman, 1991; Duncan & Humphreys, 1989; Treisman & Gormican, 1988) and the difficulty of the response transformation increased (Everling, Dorris, Klein, & Munoz, 1999; Fischer, Deubel, Wohlschlager, & Schneider, 1999; Fischer & Weber, 1992). Neurally, we found a double dissociation in posterior cortex: Brain activity in ventral regions (occipito-temporal cortex) increased linearly with perceptual, but not action, selection demands, whereas activity in dorsal regions (PPC) increased linearly with action, but not perceptual, selection demands (Figure 2B). In pFC, no double dissociation was observed; rather, activity increased only as action selection demands increased (Figure 2C). Below, we relate these findings to previous studies and discuss their implications.
Activation in occipito-temporal regions increased linearly as the set size increased during visual search. The difficulty of saccade transformation had no effect. The present task required the deployment of attention to extract the target (letter “T”) from distractors (letters “L”). We conclude that the activation in posterior ventral areas with increasing set size reflects the individuation of items, which is a critical component of the visual selection process. In this regard, increased activity reflects either the number of selection operations and/or the time required for selection to take place. Occipito-temporal regions are active during the processing of object categories or features, such as faces (Downing, Liu, & Kanwisher, 2001; O'Craven, Downing, & Kanwisher, 1999) color, and texture (Cant, Large, McCall, & Goodale, 2008; Cant & Goodale, 2007), and activation in these areas increases when attention is directed to that feature (Downing et al., 2001; O'Craven et al., 1999; Wojciulik, Kanwisher, & Driver, 1998). Furthermore, lesions of human posterior ventral areas disrupt search tasks (Humphreys, Riddoch, & Quinlan, 1985). Imaging studies have shown that word and letter recognition activate the more medial portions of the fusiform gyrus (Joseph, Cerullo, Farley, Steinmetz, & Mier, 2006; James, James, Jobard, Wong, & Gauthier, 2005; Cohen et al., 2002; Dehaene, Le Clec'H, Poline, Le Bihan, & Cohen, 2002; Polk et al., 2002), near where we found activation along the CoS. Although we did not run a functional localizer for the lateral occipital area, we note that linear effects were not observed in this region for the set size or saccade transformation contrasts (Figure 2C). The lack of lateral occipital activation in these contrasts was expected because the manipulations of set size and saccade transformation were embedded within the same visual display, which contained essentially the same number, type, and arrangement of objects.
Traditionally, the set size effect in visual search refers to the linear increase in RT as the number of distractors increases. Neuroimaging studies have not agreed on which brain areas (e.g., prefrontal, parietal, or occipital) show an effect of set size (Anderson et al., 2007; Muller et al., 2003; Leonards, Sunaert, Van Hecke, & Orban, 2000). In monkey electrophysiological studies, firing rates in both FEF (Cohen, Heitz, Woodman, & Schall, 2009) and lateral intraparietal area (Balan, Oristaglio, Schneider, & Gottlieb, 2008) neurons correlate with set size, although there is debate about the interpretation of these results (Balan & Gottlieb, 2009; Cohen et al., 2009). A consistent finding in these studies is that neuron firing rates decline proportionately as the set size increases, presumably because of competitive interactions produced by more stimuli in the stimulus array.
When interpreting past studies, one must consider two potential confounds in manipulating set size by increasing the number of distractors. First, greater set sizes in traditional visual search tasks contain more visual stimuli in the display, which would likely drive neural activity in brain areas involved in low-level visual processing. And second, increasing the number of items in the display might result in increased competitive interactions among neurons for which the potential targets lie within their receptive fields (Schall, Hanes, Thompson, & King, 1995). Therefore, it remains unclear whether processes related to attentional selection or low-level perception cause the neural changes that have been associated with increasing set size. We avoided these concerns by using placeholders in the smaller set sizes, which equated the amount of retinal stimulation across set sizes. This design allowed us to manipulate the amount of relevant visual information in the task and therefore the perceptual selection demands, without covarying the overall amount of sensory information that fell within the receptive fields of individual neurons. Importantly, our saccadic RT data show a classical set size effect, although 12 items were always present, namely, a linear increase in RT as the target-to-distractor ratio increased. Furthermore, the fact that we observed a robust set size effect in occipital and temporal regions indicates that our experiment taxed perceptual processing, and the significant correlations between saccadic RT and BOLD activation in occipital cortex, PPC, and pFC indicate that neural activation reflected the specific demands of the task (Figure 3).
Given the involvement of the PPC in the control of attention, one might expect to observe a set size effect in this region because the demand on visual attention presumably increases with set size. For example, it is reasonable to think that the number and direction (Jerde et al., 2008; Gourtzelidis et al., 2005) of covert shifts of attention would increase with the number of potential targets, and load-dependent increases have been observed in PPC for attention and working memory (Mayer et al., 2007). However, we did not find a set size effect in PPC. One possible explanation is that parietal activation might have saturated at Set Size 4 (Mitchell & Cusack, 2008). Another possibility is that visual search tasks have limited utility in testing claims about the neural effects of covert shifts of attention (Wardak, Ibos, Duhamel, & Olivier, 2006; Wardak, Olivier, & Duhamel, 2004) or claims about serial versus parallel processing of spatial attention (Leonards et al., 2000) because it is impossible to reliably measure the occurrence, timing, or direction of attention shifts during search. Indeed, in the present experiment, it is plausible that the number of covert attention shifts may not have differed appreciably across set sizes of 4, 8, and 12 potential targets. For each set size, target search may only have required attention shifts to quadrants of the stimulus array, whereupon the target is identified rather than to each item until the target is found. Because we have no evidence that increasing set sizes requires more attention shifts, the lack of a set size effect in PPC may not be surprising.
The right IPL was uniquely deactivated during the task (Figure 2A). The right TPJ, including the right IPL, is involved in filtering distractors during visual search (Shulman, Astafiev, McAvoy, d'Avossa, & Corbetta, 2007). Shulman et al. (2007) reported that the degree of deactivation was correlated with performance during visual search, such that deactivation was greater when the target was detected. In the current task, deactivation was larger in Set Size 8 compared with Set Size 4 conditions. This result suggests that the right IPL plays a role in enhancing the search process by filtering sensory information from distractors.
As the demands on action selection, but not perceptual selection, increased, neural activity increased in PPC and pFC (Figure 2B and C). The greater BOLD response in the PPC and pFC for antisaccades compared with prosaccades is consistent with past human imaging studies (Curtis & Connolly, 2008; Brown, Vilis, & Everling, 2007; Curtis & D'Esposito, 2003a, 2003b; DeSouza, Menon, & Everling, 2003). During antisaccade trials, subjects invert the saccade vector by 180° and execute a volitional saccade to the mirrored location. Rotation saccades are more demanding than antisaccades because subjects must calculate the spatial coordinates of the 90° rotation, as well as the direction of the rotation and then execute a saccade to the internalized location (Fischer et al., 1999). The activation in PPC and pFC as a function of action selection indicates that these regions are involved in the spatial–motor transformation, a conclusion that is consistent with previous studies of sensorimotor integration in parietal cortex (Buneo & Andersen, 2006; Andersen & Buneo, 2002) and with the proposed role of the lateral pFC in action selection (Miller & Cohen, 2001; Rowe, Toni, Josephs, Frackowiak, & Passingham, 2000; Wise, Boussaoud, Johnson, & Caminiti, 1997; Passingham, 1993).
Previous studies have tested theories about the functional differences of the ventral and dorsal pFC, mainly using working memory tasks (Curtis & D'Esposito, 2004; D'Esposito et al., 2000; Goldman-Rakic, 1996). Invariably, these studies probed spatial working memory with memory-guided delayed response tasks and object working memory with delayed matching- or nonmatching-to-sample tasks. Motor factors confound the spatial–object distinction because the subject knows the forthcoming motor response throughout the delay of “spatial” delayed response tasks, but not during “object” delayed matching- and nonmatching-to-sample tasks. In fact, most theories of pFC consider only stimulus features and various forms of cognitive control, but not motor factors. Given the largely segregated inputs to VLPFC and DLPFC from temporal and parietal cortex (Averbeck & Seo, 2008; Macko et al., 1982), it is rather surprising that no one to our knowledge has systematically examined the hypothesis that the human VLPFC and DLPFC are selectively engaged in the selection of perception and action, respectively. Rather, past studies of the distinction between ventral and dorsal pathways have focused almost exclusively on the posterior cortex (Cavina-Pratesi et al., 2007; Culham et al., 2003). In the lateral pFC, recent data and models have put forward a hierarchical view of its functional organization (Badre, 2008; Botvinick, 2008; Koechlin & Summerfield, 2007; Koechlin et al., 2003), with anterior regions controlling more abstract or temporally distant characteristics of cognition and behavior, whereas posterior regions control more immediate and sensorimotor characteristics. However, these models make no claims concerning the functional specialization of the ventral and dorsal regions along the anterior–posterior axis of pFC (Koechlin & Summerfield, 2007), highlighting the importance of the findings of the current study.
By imaging the human brain during a demanding visual search task, we provide the first clear neuroimaging evidence of a functional double dissociation between posterior ventral and dorsal pathways for perceptual selection and action selection, respectively (Milner & Goodale, 1995, 2008; Goodale & Milner, 1992). Despite largely segregated anatomical projections from posterior ventral and dorsal streams to lateral pFC, we did not find evidence for a functional dissociation between perception and action selection in pFC. Increasing action, but not perceptual, selection demands evoked increased activation along both the dorsal and the ventral lateral pFC. These data suggest that understanding the computations underlying action selection will be key to understanding the functional organization of pFC.
We thank the Center for Brain Imaging at New York University and their staff for support during data collection and David Heeger, Justin Gardner, and Eli Merriam for help with retinotopic mapping procedures. This work was supported by NIH R01 EY016407 to CEC and by NIH NRSA F32 EY019221 to TAJ.
Reprint requests should be sent to Clayton E. Curtis, Department of Psychology, New York University, 6 Washington Place, New York, NY 10003, USA, or via e-mail: firstname.lastname@example.org, Web: http://clayspace.psych.nyu.edu.