Older adults often encounter difficulties in switching between tasks, perhaps because of age-related decreases in executive function. Executive function may largely depend on connections between brain areas—connections that may become structurally and functionally weaker in aging. Here we investigated functional and structural age-related changes in switching between a spatial and a verbal task. These tasks were chosen because they are expected to differentially use the two hemispheres. Brain measures included anatomical information about anterior corpus callosum size (CC; the major commissure linking the left and right hemisphere), and the event-related optical signal (EROS). Behavioral results indicated that older adults had greater task-switching difficulties, which, however, were largely restricted to switching to the spatial task and to individuals with smaller anterior CCs. The EROS data showed both general switching-related activity in the left middle frontal gyrus (with approximately 300-msec latency) and task-specific activity in the inferior frontal gyrus, lateralized to the left for the switch-to-verbal condition and to the right for the switch-to-spatial condition. This lateralization was most evident in younger adults. In older adults, activity in the switch-to-spatial condition was lateralized to the right hemisphere in individuals with large CC, and to the left in individuals with small CC. These data suggest that (a) task switching may involve both task-general and task-specific processes; and (b) white matter changes may underlie some of the age-related problems in switching. These effects are discussed in terms of the hypothesis that aging involves some degree of cortical disconnection, both functional and anatomical.