We carried out an fMRI study with a twofold purpose: to investigate the relationship between networks dedicated to semantic and visual processing and to address the issue of whether semantic memory is subserved by a unique network or by different subsystems, according to semantic category or feature type. To achieve our goals, we administered a word–picture matching task, with within-category foils, to 15 healthy subjects during scanning. Semantic distance between the target and the foil and semantic domain of the target–foil pairs were varied orthogonally. Our results suggest that an amodal, undifferentiated network for the semantic processing of living things and artifacts is located in the anterolateral aspects of the temporal lobes; in fact, activity in this substrate was driven by semantic distance, not by semantic category. By contrast, activity in ventral occipito-temporal cortex was driven by category, not by semantic distance. We interpret the latter finding as the effect exerted by systematic differences between living things and artifacts at the level of their structural representations and possibly of their lower-level visual features. Finally, we attempt to reconcile contrasting data in the neuropsychological and functional imaging literature on semantic substrate and category specificity.