Deficits in visuospatial attention are commonly observed after different kinds of brain lesions. However, the structure–function relationships are not well understood. We investigated whether our response time (RT) model, strategies of visual search (STRAVIS), combined with a linear model of brain lesions, enables us to relate specific impairments in cognitive processes to specific sites of focal brain lesions. In STRAVIS, RTs in overt visual feature search with graded target-distractor similarity are decomposed into the durations of successive search steps. Fitting the model to an observer's RTs yields individual estimates of the parameters “attentional focus size,” “attentional dwell time,” and “movement time of attention or the eyes.” In 28 patients with various focal lesions to the frontal, parietal, and/or temporal cortex and 28 matched controls, we determined with the help of linear models which lesions were most predictive for each parameter. Predictions were validated with a second sample of 12 patients and 12 controls. Critical lesion areas for the STRAVIS focus size were the dorsolateral prefrontal cortex and the temporal lobe, with dorsolateral prefrontal cortex lesions reducing the focus and temporal lesions enlarging it. The STRAVIS dwell time was reduced in patients with lesions to the anterior insula and the superior parietal lobe. Lesions to the frontal eye fields, the superior parietal lobe, and the parieto-occipital cortex were most detrimental to the STRAVIS movement time. Applying linear models to a patient sample with heterogeneous lesions may be a promising new method for investigating how different brain areas interplay in a complex task.