Ventral visual cortex exhibits highly organized and selective patterns of functional activity associated with visual processing. However, this specialization decreases in normal aging, with functional responses to different visual stimuli becoming more similar with age, a phenomenon termed “dedifferentiation.” The current study tested the hypothesis that age-related degradation of the inferior longitudinal fasciculus (ILF), a white matter pathway involved in visual perception, could account for dedifferentiation of both localized and distributed brain activity in ventral visual cortex. Participants included 281 adults, ages 20–89 years, from the Dallas Lifespan Brain Study who underwent diffusion-weighted imaging to measure white matter diffusivity, as well as fMRI to measure functional selectivity to viewing photographs from different categories (e.g., faces, houses). In general, decreased ILF anisotropy significantly predicted both focal and broad functional dedifferentiation. Specifically, there was a localized effect of structure on function, such that decreased anisotropy in a smaller mid-fusiform region of ILF predicted less selective (i.e., more dedifferentiated) response to viewing faces in a proximal face-responsive region of fusiform. On the other hand, the whole ILF predicted less selective response across broader ventral visual cortex for viewing animate (e.g., human faces, animals) versus inanimate (e.g., houses, chairs) images. This structure–function relationship became weaker with age and was no longer significant after the age of 70 years. These findings indicate that decreased white matter anisotropy is associated with maladaptive differences in proximal brain function and is an important variable to consider when interpreting age differences in functional selectivity.