Aging is associated with episodic memory decline and changes in functional brain connectivity. Understanding whether and how biological sex influences age- and memory performance-related functional connectivity has important theoretical implications for the cognitive neuroscience of memory and aging. Here, we scanned 161 healthy adults between 19 and 76 years of age in an event-related fMRI study of face–location spatial context memory. Adults were scanned while performing easy and difficult versions of the task at both encoding and retrieval. We used multivariate whole-brain partial least squares connectivity to test the hypothesis that there are sex differences in age- and episodic memory performance-related functional connectivity. We examined how individual differences in age and retrieval accuracy correlated with task-related connectivity. We then repeated this analysis after disaggregating the data by self-reported sex. We found that increased encoding and retrieval-related connectivity within the dorsal attention network (DAN), and between DAN and frontoparietal network and visual networks, were positively correlated to retrieval accuracy and negatively correlated with age in both sexes. We also observed sex differences in age- and performance-related functional connectivity: (a) Greater between-networks integration was apparent at both levels of task difficulty in women only, and (b) increased DAN–default mode network connectivity with age was observed in men and was correlated with poorer memory performance. Therefore, the neural correlates of age-related episodic memory decline differ in women and men and have important theoretical and clinical implications for the cognitive neuroscience of memory, aging, and dementia prevention.