We test theories about the functional organization of the human cortex by correlating brain activity with demands on perception versus action selection. Subjects covertly searched for a target among an array of 4, 8, or 12 items (perceptual manipulation) and then, depending on the color of the array, made a saccade toward, away from, or at a right angle from the target (action manipulation). First, choice response times increased linearly as the demands increased for each factor, and brain activity in several cortical areas increased with increasing choice response times. Second, we found a double dissociation in posterior cortex: Activity in ventral regions (occipito-temporal cortex) increased linearly with perceptual, but not action, selection demands; conversely, activity in dorsal regions (parietal cortex) increased linearly with action, but not perceptual, selection demands. This result provides the clearest support of the theory that posterior cortex is segregated into two distinct streams of visual processing for perception and action. Third, despite segregated anatomical projections from posterior ventral and dorsal streams to lateral pFC, we did not find evidence for a functional dissociation between perception and action selection in pFC. Increasing action, but not perceptual, selection demands evoked increased activation along both the dorsal and the ventral lateral pFC. Although most previous studies have focused on perceptual variables (e.g., space vs. object), these data suggest that understanding the computations underlying action selection will be key to understanding the functional organization of pFC.