Perivascular spaces (PVS) play a critical role in fluid transfer and waste clearance in the brain, but few studies have explored how alterations to perivascular fluid flow may impact brain maturation and behavior across the lifespan. This study aims to characterize age-related alterations to perivascular and parenchymal fluid flow characteristics across the lifespan in typically developing children (8–21 years) and aging adults (35–90 years) and assess their contribution to cognition. In this study, we employ multi-compartment diffusion models, neurite orientation dispersion and density imaging (NODDI) and tissue tensor imaging (TTI), to quantify free water diffusion characteristics within automatically defined perivascular spaces, the surrounding parenchyma, and at variable distances from the PVS. Our findings show free water diffusion characteristics within the PVS and surrounding parenchyma are associated with age in both developing children and in aging adults. Additionally, age was associated with accelerated change in free water diffusion measures with distance from the PVS. There was no direct effect of free water diffusion measures on cognitive scores across subjects; however, a more complex relationship emerged such that age modified the relationship between free water diffusion measures and cognition. Together, these findings provide evidence of age-associated alterations to fluid flow dynamics and cognition that may be related to the waste clearance system.