An age-related decline in face processing, even under conditions in which learning and memory are not implicated, has been well documented, but the mechanism underlying this perceptual alteration remains unknown. Here, we examine whether this behavioral change may be accounted for by a reduction in white matter connectivity with age. To this end, we acquired diffusion tensor imaging data from 28 individuals aged 18 to 86 years and quantified the number of fibers, voxels, and fractional anisotropy of the two major tracts that pass through the fusiform gyrus, the pre-eminent face processing region in the ventral temporal cortex. We also measured the ability of a subset of these individuals to make fine-grained discriminations between pairs of faces and between pairs of cars. There was a significant reduction in the structural integrity of the inferior fronto-occipital fasciculus (IFOF) in the right hemisphere as a function of age on all dependent measures and there were also some changes in the left hemisphere, albeit to a lesser extent. There was also a clear age-related decrement in accuracy of perceptual discrimination, especially for more challenging perceptual discriminations, and this held to a greater degree for faces than for cars. Of greatest relevance, there was a robust association between the reduction of IFOF integrity in the right hemisphere and the decline in face perception, suggesting that the alteration in structural connectivity between the right ventral temporal and frontal cortices may account for the age-related difficulties in face processing.