Goal-relevant information can be maintained in working memory over a brief delay interval to guide an upcoming decision. There is also evidence suggesting the existence of a complementary process: namely, the ability to suppress information that is no longer relevant to ongoing task goals. Moreover, this ability to suppress or inhibit irrelevant information appears to decline with age. In this study, we compared younger and older adults undergoing fMRI on a working memory task designed to address whether the modulation of neural representations of relevant and no-longer-relevant items during a delay interval is related to age and overall task performance. Following from the theoretical predictions of the inhibitory deficit hypothesis of aging, we hypothesized that older adults would show higher activation of no-longer-relevant items during a retention delay compared to young adults and that higher activation of these no-longer-relevant items would predict worse recognition memory accuracy for relevant items. Our results support this prediction and more generally demonstrate the importance of goal-driven modulation of neural activity in successful working memory maintenance. Furthermore, we showed that the largest age differences in the regulation of category-specific pattern activity during working memory maintenance were seen throughout the medial temporal lobe and prominently in the hippocampus, further establishing the importance of “long-term memory” retrieval mechanisms in the context of high-load working memory tasks that place large demands on attentional selection mechanisms.