Despite the fact that humans are a highly social species, we know relatively little about how people remember the rich interpersonal information filling their social lives. This gap is surprising: One function of memory has been suggested to be relationship maintenance (Neisser, 1988). A major barrier to understanding the brain basis of interpersonal memory is that traditional brain imaging methods are not ideally suited to study memory for the nuanced interpersonal experiences comprising our social lives. Yet, recent and rapidly developing advances in the analysis of brain responses to naturalistic social information can help researchers surpass this methodological barrier. This perspective piece articulates the importance of studying the brain basis of real-world social memories and suggests new directions in interpersonal memory research. This includes investigating the brain mechanisms that represent the content and structure of real-world interpersonal memories as well as how they are altered in mental health conditions associated with social memory biases.