Aging is associated with episodic memory decline and alterations in memory-related brain function. However, it remains unclear if age-related memory decline is associated with similar patterns of brain aging in women and men. In the current task fMRI study, we tested the hypothesis that there are sex differences in the effect of age and memory performance on brain activity during episodic encoding and retrieval of face–location associations (spatial context memory). Forty-one women and 41 men between the ages of 21 and 76 years participated in this study. Between-group multivariate partial least squares analysis of the fMRI data was conducted to directly test for sex differences and similarities in age-related and performance-related patterns of brain activity. Our behavioral analysis indicated no significant sex differences in retrieval accuracy on the fMRI tasks. In relation to performance effects, we observed similarities and differences in how retrieval accuracy related to brain activity in women and men. Both sexes activated dorsal and lateral PFC, inferior parietal cortex, and left parahippocampal gyrus at encoding, and this supported subsequent memory performance. However, there were sex differences in retrieval activity in these same regions and in lateral occipital-temporal and ventrolateral PFC. In relation to age effects, we observed sex differences in the effect of age on memory-related activity within PFC, inferior parietal cortex, parahippocampal gyrus, and lateral occipital-temporal cortices. Overall, our findings suggest that the neural correlates of age-related spatial context memory decline differ in women compared with men.