Previous research has established that age-related decline occurs in measures of cerebral white matter integrity, but the role of this decline in age-related cognitive changes is not clear. To conclude that white matter integrity has a mediating (causal) contribution, it is necessary to demonstrate that statistical control of the white matter–cognition relation reduces the magnitude of age–cognition relation. In this research, we tested the mediating role of white matter integrity, in the context of a task-switching paradigm involving word categorization. Participants were 20 healthy, community-dwelling older adults (60–85 years), and 20 younger adults (18–27 years). From diffusion tensor imaging tractography, we obtained fractional anisotropy (FA) as an index of white matter integrity in the genu and splenium of the corpus callosum and the superior longitudinal fasciculus (SLF). Mean FA values exhibited age-related decline consistent with a decrease in white matter integrity. From a model of reaction time distributions, we obtained independent estimates of the decisional and nondecisional (perceptual–motor) components of task performance. Age-related decline was evident in both components. Critically, age differences in task performance were mediated by FA in two regions: the central portion of the genu, and splenium–parietal fibers in the right hemisphere. This relation held only for the decisional component and was not evident in the nondecisional component. This result is the first demonstration that the integrity of specific white matter tracts is a mediator of age-related changes in cognitive performance.