Humans are able to use nonverbal behavior to make fast, reliable judgments of both emotional states and personality traits. Whereas a sizeable body of research has identified neural structures critical for emotion recognition, the neural substrates of personality trait attribution have not been explored in detail. In the present study, we investigated the neural systems involved in emotion and personality trait judgments. We used a type of visual stimulus that is known to convey both emotion and personality information, namely, point-light walkers. We compared the emotion and personality trait judgments made by subjects with brain damage to those made by neurologically normal subjects and then conducted a lesion overlap analysis to identify neural regions critical for these two tasks. Impairments on the two tasks dissociated: Some subjects were impaired at emotion recognition, but judged personality normally; other subjects were impaired on the personality task, but normal at emotion recognition. Moreover, these dissociations in performance were associated with damage to specific neural regions: Right somatosensory cortices were a primary focus of lesion overlap in subjects impaired on the emotion task, whereas left frontal opercular cortices were a primary focus of lesion overlap in subjects impaired on the personality task. These findings suggest that attributions of emotional states and personality traits are accomplished by partially dissociable neural systems.