Neuropsychological studies suggest that knowledge about living and nonliving objects is processed in separate brain regions. However, lesion and functional neuroimaging studies have implicated different areas. To address this issue, we used voxel-based morphometry to correlate accuracy in naming line drawings of living and nonliving objects with gray matter volumes in 152 patients with various neurodegenerative diseases. The results showed a significant positive correlation between gray matter volumes in bilateral temporal cortices and total naming accuracy regardless of category. Naming scores for living stimuli correlated with gray matter volume in the medial portion of the right anterior temporal pole, whereas naming accuracy for familiarity-matched nonliving items correlated with the volume of the left posterior middle temporal gyrus. A previous behavioral study showed that the living stimuli used here also had in common the characteristic that they were defined by shared sensory semantic features, whereas items in the nonliving group were defined by their action-related semantic features. We propose that the anatomical segregation of living and nonliving categories is the result of their defining semantic features and the distinct neural subsystems used to process them.