Altered functional connectivity is related to severity of language impairment in poststroke aphasia. However, it is not clear whether this finding specifically reflects loss of functional coherence, or more generally, is related to decreased structural connectivity due to cortical necrosis. The aim of the current study was to investigate this issue by factoring out structural connectivity from functional connectivity measures and then relating the residual data to language performance poststroke. Ninety-seven participants with a history of stroke were assessed using language impairment measures ( Auditory Verbal Comprehension and Spontaneous Speech scores from the Western Aphasia Battery–Revised) and MRI (structural, diffusion tensor imaging, and resting-state functional connectivity). We analyzed the association between functional connectivity and language and controlled for multiple potential neuroanatomical confounders, namely structural connectivity. We identified functional connections within the left hemisphere ventral stream where decreased functional connectivity, independent of structural connectivity, was associated with speech comprehension impairment. These connections exist in frontotemporal and temporoparietal regions. Our results suggest poor speech comprehension in aphasia is at least partially caused by loss of cortical synchrony in a left hemisphere ventral stream network and is not only reflective of localized necrosis or structural connectivity.