Because of our limited processing capacity, different elements of the visual scene compete for the allocation of processing resources. One of the most striking deficits in visual selection is simultanagnosia, a rare neuropsychological condition characterized by impaired spatial awareness of more than one object at time. To decompose the neuroanatomical substrates of the syndrome and to gain insights into the structural and functional organization of visuospatial attention, we performed a systematic evaluation of lesion patterns in a group of simultanagnosic patients compared with patients with either (i) unilateral visuospatial deficits (neglect and/or extinction) or (ii) bilateral posterior lesions without visuospatial deficits, using overlap/subtraction analyses, estimation of lesion volume, and a lesion laterality index. We next used voxel-based morphometry to assess the link between different visuospatial deficits and gray matter and white matter (WM) damage. Lesion overlap/subtraction analyses, lesion laterality index, and voxel-based morphometry measures converged to indicate that bilateral parieto-occipital WM disconnections are both distinctive and necessary to create symptoms associated with simultanagnosia. We also found that bilateral gray matter damage within the middle frontal area (BA 46), cuneus, calacarine, and parieto-occipital fissure as well as right hemisphere parietal lesions within intraparietal and postcentral gyri were associated with simultanagnosia. Further analysis of the WM based on tractography revealed associations with bilateral damage to major pathways within the visuospatial attention network, including the superior longitudinal fasciculus, the inferior fronto-occipital fasciculus, and the inferior longitudinal fasciculus. We conclude that damage to the parieto-occipital regions and the intraparietal sulcus, together, with bilateral WM disconnections within the visuosptial attention network, contribute to poor visual processing of multiple objects and the loss of processing speed characteristic of simultanagnosia.