Different categories of visual objects evoke distinct stimulus-evoked sensory responses in extrastriate visual cortex. Although numerous lines of evidence support a distinct representational neural architecture, the mechanisms underlying the modulation of the category selectivity by top–down influences remains uncertain. In this study, we investigate the causal role of the PFC in the modulation of evoked activity to face and scene stimuli in the extrastriate cortex. We used two experimental approaches to disrupt prefrontal cortical function—repetitive TMS to PFC in healthy participants (Experiment 1) and focal PFC lesions in stroke patients (Experiment 2). After these perturbations to normal PFC function (pre- vs. post-TMS and lesion vs. intact hemisphere), stimulus-evoked activity in extrastriate cortex exhibited less distinct category selectivity to faces and scenes. These two experiments provide convergent evidence highlighting a direct role of PFC in the top–down modulation of bottom–up visual signals.