Goal-irrelevant information may interfere with ongoing task activities if not controlled properly. Evidence suggests that the ability to control interference is connected mainly to the prefrontal cortex (pFC). However, it remains unclear whether gray matter (GM) volume in prefrontal regions influences individual differences in interference control (IC) and if these relationships are affected by aging. Using cross-sectional and longitudinal estimates over a 4- to 5-year period, we examined the relationship between relative IC scores, obtained from a 2-back working memory task, GM volumes, and performance in different cognitive domains. By identifying individuals with either no or high levels of interference, we demonstrated that participants with superior IC had larger volume of the ventrolateral pFC, regardless of participant demographics. The same pattern was observed both at baseline and follow-up. Cross-sectional estimates further showed that interference increased as a function of age, but interference did not change between baseline and follow-up. Similarly, across-sample associations between IC and pFC volume were found in the cross-sectional data, along with no longitudinal change–change relationships. Moreover, relative IC scores could be linked to composite scores of fluid intelligence, indicating that control of interference may relate to performance in expected cognitive domains. These results provide new evidence that a relative IC score can be related to volume of specific and relevant regions within pFC and that this relationship is not modulated by age. This supports a view that the GM volume in these regions plays a role in resisting interference during a working memory task.