Age has a differential effect on cognition, with word retrieval being one of the cognitive domains most affected by aging. This study examined the functional and structural neural correlates of phonological word retrieval in younger and older adults using word and picture rhyme judgment tasks. Although the behavioral performance in the fMRI task was similar for the two age groups, the older adults had increased activation in the right pars triangularis across tasks and in the right pars orbitalis for the word task only. Increased activation together with preserved performance in the older participants would suggest that increased activation was related to compensatory processing. We validated this hypothesis by showing that right pars triangularis activation during correct rhyme judgments was highest in participants who made overall more errors, therefore being most error-prone. Our findings demonstrate that the effect of aging differ in adjacent but distinct right inferior frontal regions. The differential effect of age on word and picture tasks also provides new clues to the level of processing that is most affected by age in speech production tasks. Specifically, we suggest that right inferior frontal activation in older participants is needed to inhibit errors.