Deep learning models highly accurately predict brain age from MRI but their explanatory capacity is limited. Explainable artificial intelligence (XAI) methods can identify relevant voxels contributing to model estimates, yet they do not reveal which biological features these voxels represent. In this study, we closed this gap by relating voxel-based contributions to brain-age estimates, extracted with XAI, to human-interpretable structural features of the aging brain. To this end, we associated participant-level XAI-based relevance maps extracted from two ensembles of 3D-convolutional neural networks (3D-CNNs) that were trained on T1-weighted and fluid-attenuated inversion recovery images of 1855 participants (age range 18–82 years), with regional cortical and subcortical gray matter volume and thickness, perivascular spaces (PVS), and water diffusion-based fractional anisotropy of major white matter tracts. We found that all neuroimaging markers of brain aging, except for PVS, were highly correlated with the XAI-based relevance maps. Overall, the strongest correlation was found between ventricular volume and relevance (r = 0.69), and by feature, temporal-parietal cortical thickness and volume, cerebellar gray matter volume, and frontal-occipital white matter tracts showed the strongest correlations with XAI-based relevance. Our ensembles of 3D-CNNs took into account a plethora of known aging processes in the brain to perform age prediction. Some age-associated features like PVS were not consistently considered by the models, and the cerebellum was more important than expected. Taken together, we highlight the ability of end-to-end deep learning models combined with XAI to reveal biologically relevant, multi-feature relationships in the brain.