Theories of word production and word recognition generally agree that multiple word candidates are activated during processing. The facilitative and inhibitory effects of these “lexical neighbors” have been studied extensively using behavioral methods and have spurred theoretical development in psycholinguistics, but relatively little is known about the neural basis of these effects and how lesions may affect them. This study used voxel-wise lesion overlap subtraction to examine semantic and phonological neighbor effects in spoken word production following left hemisphere stroke. Increased inhibitory effects of near semantic neighbors were associated with inferior frontal lobe lesions, suggesting impaired selection among strongly activated semantically related candidates. Increased inhibitory effects of phonological neighbors were associated with posterior superior temporal and inferior parietal lobe lesions. In combination with previous studies, these results suggest that such lesions cause phonological-to-lexical feedback to more strongly activate phonologically related lexical candidates. The comparison of semantic and phonological neighbor effects and how they are affected by left hemisphere lesions provides new insights into the cognitive dynamics and neural basis of phonological, semantic, and cognitive control processes in spoken word production.