Functional neuroimaging has highlighted a left-hemisphere conceptual system shared by verbal and nonverbal processing despite neuropsychological evidence that the ability to recognize verbal and nonverbal stimuli can doubly dissociate in patients with left- and right-hemisphere lesions, respectively. Previous attempts to control for perceptual differences between verbal and nonverbal stimuli in functional neuroimaging studies may have hidden differences arising at the conceptual level. Here we used a different approach and controlled for perceptual confounds by looking for amodal verbal and nonverbal conceptual activations that are common to both the visual and auditory modalities. In addition to the left-hemisphere conceptual system activated by all meaningful stimuli, we observed the left/right double dissociation in verbal and nonverbal conceptual processing, predicted by neuropsychological studies. Left middle and superior temporal regions were selectively more involved in comprehending words—heard or read—and the right midfusiform and right posterior middle temporal cortex were selectively more involved in making sense of environmental sounds and images. Thus, the neuroanatomical basis of a verbal/nonverbal conceptual processing dissociation is established.