Perception and understanding of dispositions and intentions of others through their actions are of immense importance for adaptive daily-life behavior and social communication. Here we ask whether, and, if so, how this ability is impaired in adolescents who were born premature and suffer early periventricular damage, periventricular leukomalacia (PVL) that affects brain connectivity. The visual event arrangement (EA) task was administered to PVL patients and two control groups, premature-born and term-born adolescents without brain abnormalities on a magnetic resonance imaging scan. Performance on the EA task was significantly lower in PVL patients as compared with controls. No difference was found between premature-born participants without lesions and term-born controls. Performance on the EA task was inversely related to the volumetric extent of lesions in the parieto-occipital regions of both hemispheres and, in particular, to the right temporal periventricular lesions. Whereas our earlier work reveals that compromised visual processing of biological motion, impairments in visual navigation, and other visual-perceptual disabilities in PVL patients are associated with parieto-occipital lesions, difficulties in the visual EA task solely are specifically linked to the right temporal periventricular lesions. For the first time, we show that the severity of the right temporal PVL can serve as a predictor of the ability for perception and understanding of others' actions. We assume that impairments in this ability in PVL patients are caused by disrupted brain connectivity to the right temporal cortex, a key node of the social brain.