Most contemporary theories of semantic memory assume that concepts are formed from the distillation of information arising in distinct sensory and verbal modalities. The neural basis of this distillation or convergence of information was the focus of this study. Specifically, we explored two commonly posed hypotheses: (a) that the human middle temporal gyrus (MTG) provides a crucial semantic interface given the fact that it interposes auditory and visual processing streams and (b) that the anterior temporal region—especially its ventral surface (vATL)—provides a critical region for the multimodal integration of information. By utilizing distortion-corrected fMRI and an established semantic association assessment (commonly used in neuropsychological investigations), we compared the activation patterns observed for both the verbal and nonverbal versions of the same task. The results are consistent with the two hypotheses simultaneously: Both MTG and vATL are activated in common for word and picture semantic processing. Additional planned, ROI analyses show that this result follows from two principal axes of convergence in the temporal lobe: both lateral (toward MTG) and longitudinal (toward the anterior temporal lobe).